Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 13:59:12.674990 28740 test_util.cc:268] Using random seed: -466724740
I20240501 13:59:12.687399 28740 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:12.687508 28740 ts_itest-base.cc:113] --------------
I20240501 13:59:12.687551 28740 ts_itest-base.cc:114] 5 tablet servers
I20240501 13:59:12.687615 28740 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:12.687677 28740 ts_itest-base.cc:116] --------------
2024-05-01T13:59:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:12Z Disabled control of system clock
I20240501 13:59:12.712284 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:32907
--webserver_interface=127.28.17.62
--webserver_port=0
--builtin_ntp_servers=127.28.17.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.17.62:32907
--enable_range_replica_placement=false with env {}
W20240501 13:59:13.054055 28752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:13.054376 28752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:13.054466 28752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:13.062062 28752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:13.062196 28752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:13.062237 28752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:13.062299 28752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:13.072090 28752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.17.62:32907
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.17.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:13.073509 28752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:13.075410 28752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:13.087604 28757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.088125 28758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.088634 28760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:13.089146 28752 server_base.cc:1034] running on GCE node
I20240501 13:59:13.089972 28752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:13.091576 28752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:13.092798 28752 hybrid_clock.cc:648] HybridClock initialized: now 1714571953092727 us; error 47 us; skew 500 ppm
I20240501 13:59:13.093147 28752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:13.098713 28752 webserver.cc:460] Webserver started at http://127.28.17.62:42159/ using document root <none> and password file <none>
I20240501 13:59:13.099496 28752 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:13.099634 28752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:13.099943 28752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:13.102834 28752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f7b3e7e00cca4d33abb0286cea4b4c22"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-dx8c"
I20240501 13:59:13.103623 28752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f7b3e7e00cca4d33abb0286cea4b4c22"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-dx8c"
I20240501 13:59:13.109206 28752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:13.112907 28766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.114396 28752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:13.114612 28752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f7b3e7e00cca4d33abb0286cea4b4c22"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-dx8c"
I20240501 13:59:13.114796 28752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:13.137859 28752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:13.138873 28752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:13.139155 28752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:13.157172 28752 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.62:32907
I20240501 13:59:13.157191 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.62:32907 every 8 connection(s)
I20240501 13:59:13.158725 28752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:13.159996 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 28752
I20240501 13:59:13.160272 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:13.162863 28818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:13.175969 28818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22: Bootstrap starting.
I20240501 13:59:13.179908 28818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:13.181318 28818 log.cc:824] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:13.184257 28818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22: No bootstrap required, opened a new log
I20240501 13:59:13.191243 28818 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 32907 } }
I20240501 13:59:13.191643 28818 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:13.191754 28818 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b3e7e00cca4d33abb0286cea4b4c22, State: Initialized, Role: FOLLOWER
I20240501 13:59:13.192374 28818 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 32907 } }
I20240501 13:59:13.192616 28818 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:13.192747 28818 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:13.192909 28818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:13.270139 28818 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 32907 } }
I20240501 13:59:13.270820 28818 leader_election.cc:302] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b3e7e00cca4d33abb0286cea4b4c22; no voters:
I20240501 13:59:13.271533 28818 leader_election.cc:288] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:13.271718 28823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:13.274003 28818 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:13.274224 28823 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [term 1 LEADER]: Becoming Leader. State: Replica: f7b3e7e00cca4d33abb0286cea4b4c22, State: Running, Role: LEADER
I20240501 13:59:13.274787 28823 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 32907 } }
I20240501 13:59:13.278173 28825 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3e7e00cca4d33abb0286cea4b4c22. Latest consensus state: current_term: 1 leader_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 32907 } } }
I20240501 13:59:13.278198 28824 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3e7e00cca4d33abb0286cea4b4c22" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 32907 } } }
I20240501 13:59:13.278653 28824 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:13.278658 28825 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:13.283892 28833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:13.288708 28833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:13.299027 28833 catalog_manager.cc:1347] Generated new cluster ID: d2ce1032cead45cd9ef97c6870ab50c4
I20240501 13:59:13.299154 28833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:13.319833 28833 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:13.321552 28833 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:13.330695 28833 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f7b3e7e00cca4d33abb0286cea4b4c22: Generated new TSK 0
I20240501 13:59:13.331636 28833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:13.344519 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.1:0
--local_ip_for_outbound_sockets=127.28.17.1
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--builtin_ntp_servers=127.28.17.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 13:59:13.681737 28842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:13.682053 28842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:13.682168 28842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:13.689999 28842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:13.690212 28842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.1
I20240501 13:59:13.699499 28842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.1
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:13.700870 28842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:13.702667 28842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:13.715281 28848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.715737 28847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.716390 28850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:13.717084 28842 server_base.cc:1034] running on GCE node
I20240501 13:59:13.717784 28842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:13.718874 28842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:13.720114 28842 hybrid_clock.cc:648] HybridClock initialized: now 1714571953720032 us; error 53 us; skew 500 ppm
I20240501 13:59:13.720533 28842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:13.722916 28842 webserver.cc:460] Webserver started at http://127.28.17.1:32795/ using document root <none> and password file <none>
I20240501 13:59:13.723690 28842 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:13.723819 28842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:13.724123 28842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:13.726934 28842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cd0cff7c8bf9440f95e69565a98979fd"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-dx8c"
I20240501 13:59:13.727726 28842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cd0cff7c8bf9440f95e69565a98979fd"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-dx8c"
I20240501 13:59:13.733215 28842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 13:59:13.737039 28856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.738613 28842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 13:59:13.738809 28842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cd0cff7c8bf9440f95e69565a98979fd"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-dx8c"
I20240501 13:59:13.739017 28842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:13.761629 28842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:13.762565 28842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:13.762898 28842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:13.763906 28842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:13.765455 28842 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:13.765533 28842 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.765671 28842 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:13.765733 28842 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.797704 28842 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.1:43985
I20240501 13:59:13.797757 28968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.1:43985 every 8 connection(s)
I20240501 13:59:13.799086 28842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:13.802457 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 28842
I20240501 13:59:13.802708 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:13.806134 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.2:0
--local_ip_for_outbound_sockets=127.28.17.2
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--builtin_ntp_servers=127.28.17.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:13.813035 28969 heartbeater.cc:347] Connected to a master server at 127.28.17.62:32907
I20240501 13:59:13.813357 28969 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:13.814213 28969 heartbeater.cc:511] Master 127.28.17.62:32907 requested a full tablet report, sending...
I20240501 13:59:13.816555 28783 ts_manager.cc:194] Registered new tserver with Master: cd0cff7c8bf9440f95e69565a98979fd (127.28.17.1:43985)
I20240501 13:59:13.818150 28783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.1:33385
W20240501 13:59:14.138485 28973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:14.138872 28973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:14.139007 28973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:14.146808 28973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:14.147022 28973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.2
I20240501 13:59:14.156313 28973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.2
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:14.157716 28973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:14.159616 28973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:14.173502 28978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.174173 28979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.174638 28981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.175100 28973 server_base.cc:1034] running on GCE node
I20240501 13:59:14.175727 28973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:14.176677 28973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:14.177891 28973 hybrid_clock.cc:648] HybridClock initialized: now 1714571954177819 us; error 46 us; skew 500 ppm
I20240501 13:59:14.178205 28973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:14.180615 28973 webserver.cc:460] Webserver started at http://127.28.17.2:41437/ using document root <none> and password file <none>
I20240501 13:59:14.181416 28973 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:14.181536 28973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:14.181852 28973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:14.184607 28973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "676557eea03049649831d6579fce3378"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-dx8c"
I20240501 13:59:14.185436 28973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "676557eea03049649831d6579fce3378"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-dx8c"
I20240501 13:59:14.191044 28973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 13:59:14.194947 28987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.196602 28973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:14.196797 28973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "676557eea03049649831d6579fce3378"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-dx8c"
I20240501 13:59:14.196993 28973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:14.252214 28973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:14.253108 28973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:14.253407 28973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:14.254474 28973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:14.256069 28973 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:14.256141 28973 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.256242 28973 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:14.256301 28973 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.287194 28973 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.2:41953
I20240501 13:59:14.287230 29099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.2:41953 every 8 connection(s)
I20240501 13:59:14.288592 28973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:14.293534 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 28973
I20240501 13:59:14.293690 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:14.297129 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.3:0
--local_ip_for_outbound_sockets=127.28.17.3
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--builtin_ntp_servers=127.28.17.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:14.302600 29100 heartbeater.cc:347] Connected to a master server at 127.28.17.62:32907
I20240501 13:59:14.302942 29100 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:14.303824 29100 heartbeater.cc:511] Master 127.28.17.62:32907 requested a full tablet report, sending...
I20240501 13:59:14.305423 28783 ts_manager.cc:194] Registered new tserver with Master: 676557eea03049649831d6579fce3378 (127.28.17.2:41953)
I20240501 13:59:14.306485 28783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.2:52523
W20240501 13:59:14.629104 29103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:14.629419 29103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:14.629514 29103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:14.637058 29103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:14.637252 29103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.3
I20240501 13:59:14.646700 29103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.3
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:14.648077 29103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:14.649798 29103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:14.666899 29111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.667316 29108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.668032 29109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.668709 29103 server_base.cc:1034] running on GCE node
I20240501 13:59:14.669422 29103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:14.670464 29103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:14.671741 29103 hybrid_clock.cc:648] HybridClock initialized: now 1714571954671662 us; error 50 us; skew 500 ppm
I20240501 13:59:14.672060 29103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:14.674563 29103 webserver.cc:460] Webserver started at http://127.28.17.3:35727/ using document root <none> and password file <none>
I20240501 13:59:14.675336 29103 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:14.675465 29103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:14.675763 29103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:14.678586 29103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "461ff760d8944577b240697a6eb0b736"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-dx8c"
I20240501 13:59:14.679467 29103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "461ff760d8944577b240697a6eb0b736"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-dx8c"
I20240501 13:59:14.685005 29103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:14.688831 29117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.690418 29103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:14.690634 29103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "461ff760d8944577b240697a6eb0b736"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-dx8c"
I20240501 13:59:14.690819 29103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:14.709009 29103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:14.709887 29103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:14.710165 29103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:14.711172 29103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:14.712898 29103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:14.712975 29103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.713078 29103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:14.713127 29103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:14.742785 29103 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.3:42361
I20240501 13:59:14.742831 29229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.3:42361 every 8 connection(s)
I20240501 13:59:14.744175 29103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:14.745438 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 29103
I20240501 13:59:14.745661 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:14.749222 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.4:0
--local_ip_for_outbound_sockets=127.28.17.4
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--builtin_ntp_servers=127.28.17.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:14.759240 29230 heartbeater.cc:347] Connected to a master server at 127.28.17.62:32907
I20240501 13:59:14.759588 29230 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:14.760444 29230 heartbeater.cc:511] Master 127.28.17.62:32907 requested a full tablet report, sending...
I20240501 13:59:14.762023 28783 ts_manager.cc:194] Registered new tserver with Master: 461ff760d8944577b240697a6eb0b736 (127.28.17.3:42361)
I20240501 13:59:14.762881 28783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.3:48759
I20240501 13:59:14.821671 28969 heartbeater.cc:503] Master 127.28.17.62:32907 was elected leader, sending a full tablet report...
W20240501 13:59:15.094126 29233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.094446 29233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.094583 29233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.102470 29233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.102728 29233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.4
I20240501 13:59:15.112025 29233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.4
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.113358 29233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.115164 29233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.128110 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.128779 29238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.129138 29241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.129638 29233 server_base.cc:1034] running on GCE node
I20240501 13:59:15.130291 29233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.131373 29233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.132598 29233 hybrid_clock.cc:648] HybridClock initialized: now 1714571955132532 us; error 47 us; skew 500 ppm
I20240501 13:59:15.132915 29233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.135341 29233 webserver.cc:460] Webserver started at http://127.28.17.4:37823/ using document root <none> and password file <none>
I20240501 13:59:15.136083 29233 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.136214 29233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.136516 29233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.139225 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "06286426acec47d1955891d8e19ccdd3"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-dx8c"
I20240501 13:59:15.140020 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "06286426acec47d1955891d8e19ccdd3"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-dx8c"
I20240501 13:59:15.145507 29233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 13:59:15.149137 29247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.150713 29233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:15.150884 29233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "06286426acec47d1955891d8e19ccdd3"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-dx8c"
I20240501 13:59:15.151072 29233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.177383 29233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.178295 29233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.178632 29233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.179592 29233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.181105 29233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.181185 29233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.181303 29233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.181378 29233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.210120 29233 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.4:37821
I20240501 13:59:15.210153 29359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.4:37821 every 8 connection(s)
I20240501 13:59:15.211477 29233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:15.218755 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 29233
I20240501 13:59:15.218950 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:15.222205 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:0
--local_ip_for_outbound_sockets=127.28.17.5
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--builtin_ntp_servers=127.28.17.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 13:59:15.225152 29360 heartbeater.cc:347] Connected to a master server at 127.28.17.62:32907
I20240501 13:59:15.225476 29360 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.226572 29360 heartbeater.cc:511] Master 127.28.17.62:32907 requested a full tablet report, sending...
I20240501 13:59:15.228178 28783 ts_manager.cc:194] Registered new tserver with Master: 06286426acec47d1955891d8e19ccdd3 (127.28.17.4:37821)
I20240501 13:59:15.229267 28783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.4:36385
I20240501 13:59:15.309363 29100 heartbeater.cc:503] Master 127.28.17.62:32907 was elected leader, sending a full tablet report...
W20240501 13:59:15.571754 29363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:15.572058 29363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:15.572178 29363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:15.580313 29363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:15.580528 29363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 13:59:15.589879 29363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:32907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:15.591302 29363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.593641 29363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.606717 29371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.607041 29368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.607606 29369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.608222 29363 server_base.cc:1034] running on GCE node
I20240501 13:59:15.608889 29363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:15.609912 29363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:15.611151 29363 hybrid_clock.cc:648] HybridClock initialized: now 1714571955611072 us; error 66 us; skew 500 ppm
I20240501 13:59:15.611491 29363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.614341 29363 webserver.cc:460] Webserver started at http://127.28.17.5:44927/ using document root <none> and password file <none>
I20240501 13:59:15.615130 29363 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.615269 29363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.615571 29363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.618435 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "78a82770776349429229de36040aa3c9"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-dx8c"
I20240501 13:59:15.619364 29363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "78a82770776349429229de36040aa3c9"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-dx8c"
I20240501 13:59:15.624959 29363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:15.628732 29377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.630307 29363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:15.630488 29363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "78a82770776349429229de36040aa3c9"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-dx8c"
I20240501 13:59:15.630712 29363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.658308 29363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.659212 29363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.659525 29363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.660466 29363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.661974 29363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.662050 29363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.662173 29363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.662223 29363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.696041 29363 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:36767
I20240501 13:59:15.696094 29489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:36767 every 8 connection(s)
I20240501 13:59:15.697919 29363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:15.700695 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 29363
I20240501 13:59:15.700894 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:15.710744 29490 heartbeater.cc:347] Connected to a master server at 127.28.17.62:32907
I20240501 13:59:15.711045 29490 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.711849 29490 heartbeater.cc:511] Master 127.28.17.62:32907 requested a full tablet report, sending...
I20240501 13:59:15.713270 28783 ts_manager.cc:194] Registered new tserver with Master: 78a82770776349429229de36040aa3c9 (127.28.17.5:36767)
I20240501 13:59:15.714025 28783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:49475
I20240501 13:59:15.717846 28740 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:15.740206 28783 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54396:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:15.765576 29230 heartbeater.cc:503] Master 127.28.17.62:32907 was elected leader, sending a full tablet report...
I20240501 13:59:15.786007 29425 tablet_service.cc:1467] Processing CreateTablet for tablet 62963132761a4c5d98ef7ff5a32f8cb6 (DEFAULT_TABLE table=TestTable [id=66e5bd360f6948cbbf5dd674be3bf1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:15.786644 29165 tablet_service.cc:1467] Processing CreateTablet for tablet 62963132761a4c5d98ef7ff5a32f8cb6 (DEFAULT_TABLE table=TestTable [id=66e5bd360f6948cbbf5dd674be3bf1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:15.786962 29295 tablet_service.cc:1467] Processing CreateTablet for tablet 62963132761a4c5d98ef7ff5a32f8cb6 (DEFAULT_TABLE table=TestTable [id=66e5bd360f6948cbbf5dd674be3bf1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:15.787628 29425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62963132761a4c5d98ef7ff5a32f8cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:15.787750 29165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62963132761a4c5d98ef7ff5a32f8cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:15.788337 29295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62963132761a4c5d98ef7ff5a32f8cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:15.805845 29508 tablet_bootstrap.cc:492] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736: Bootstrap starting.
I20240501 13:59:15.807760 29509 tablet_bootstrap.cc:492] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3: Bootstrap starting.
I20240501 13:59:15.809861 29510 tablet_bootstrap.cc:492] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9: Bootstrap starting.
I20240501 13:59:15.811007 29508 tablet_bootstrap.cc:654] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:15.811439 29509 tablet_bootstrap.cc:654] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:15.812057 29508 log.cc:824] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:15.812621 29509 log.cc:824] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:15.814016 29510 tablet_bootstrap.cc:654] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:15.815332 29510 log.cc:824] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:15.816998 29509 tablet_bootstrap.cc:492] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3: No bootstrap required, opened a new log
I20240501 13:59:15.816998 29508 tablet_bootstrap.cc:492] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736: No bootstrap required, opened a new log
I20240501 13:59:15.817296 29508 ts_tablet_manager.cc:1367] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240501 13:59:15.817305 29509 ts_tablet_manager.cc:1367] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240501 13:59:15.817555 29510 tablet_bootstrap.cc:492] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9: No bootstrap required, opened a new log
I20240501 13:59:15.817800 29510 ts_tablet_manager.cc:1367] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 13:59:15.824048 29510 raft_consensus.cc:348] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.824534 29510 raft_consensus.cc:374] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:15.825138 29510 raft_consensus.cc:729] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a82770776349429229de36040aa3c9, State: Initialized, Role: FOLLOWER
I20240501 13:59:15.825970 29510 consensus_queue.cc:260] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.826293 29509 raft_consensus.cc:348] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.826766 29509 raft_consensus.cc:374] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:15.826897 29509 raft_consensus.cc:729] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06286426acec47d1955891d8e19ccdd3, State: Initialized, Role: FOLLOWER
I20240501 13:59:15.827592 29509 consensus_queue.cc:260] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.827492 29508 raft_consensus.cc:348] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.827913 29508 raft_consensus.cc:374] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:15.828027 29508 raft_consensus.cc:729] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461ff760d8944577b240697a6eb0b736, State: Initialized, Role: FOLLOWER
I20240501 13:59:15.828791 29508 consensus_queue.cc:260] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.830835 29508 ts_tablet_manager.cc:1398] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20240501 13:59:15.833724 29510 ts_tablet_manager.cc:1398] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20240501 13:59:15.834662 29360 heartbeater.cc:503] Master 127.28.17.62:32907 was elected leader, sending a full tablet report...
I20240501 13:59:15.834744 29490 heartbeater.cc:503] Master 127.28.17.62:32907 was elected leader, sending a full tablet report...
I20240501 13:59:15.835400 29509 ts_tablet_manager.cc:1398] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3: Time spent starting tablet: real 0.018s user 0.013s sys 0.007s
I20240501 13:59:15.907310 29514 raft_consensus.cc:482] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:15.907601 29514 raft_consensus.cc:504] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.909404 29514 leader_election.cc:288] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 461ff760d8944577b240697a6eb0b736 (127.28.17.3:42361), 06286426acec47d1955891d8e19ccdd3 (127.28.17.4:37821)
I20240501 13:59:15.916625 29315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62963132761a4c5d98ef7ff5a32f8cb6" candidate_uuid: "78a82770776349429229de36040aa3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06286426acec47d1955891d8e19ccdd3" is_pre_election: true
I20240501 13:59:15.917184 29315 raft_consensus.cc:2454] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a82770776349429229de36040aa3c9 in term 0.
I20240501 13:59:15.917851 29381 leader_election.cc:302] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06286426acec47d1955891d8e19ccdd3, 78a82770776349429229de36040aa3c9; no voters:
I20240501 13:59:15.917989 29185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62963132761a4c5d98ef7ff5a32f8cb6" candidate_uuid: "78a82770776349429229de36040aa3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461ff760d8944577b240697a6eb0b736" is_pre_election: true
I20240501 13:59:15.918217 29514 raft_consensus.cc:2781] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:15.918380 29514 raft_consensus.cc:482] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:15.918478 29514 raft_consensus.cc:3037] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:15.918537 29185 raft_consensus.cc:2454] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a82770776349429229de36040aa3c9 in term 0.
I20240501 13:59:15.921249 29514 raft_consensus.cc:504] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.921837 29514 leader_election.cc:288] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [CANDIDATE]: Term 1 election: Requested vote from peers 461ff760d8944577b240697a6eb0b736 (127.28.17.3:42361), 06286426acec47d1955891d8e19ccdd3 (127.28.17.4:37821)
I20240501 13:59:15.922271 29185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62963132761a4c5d98ef7ff5a32f8cb6" candidate_uuid: "78a82770776349429229de36040aa3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461ff760d8944577b240697a6eb0b736"
I20240501 13:59:15.922400 29315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62963132761a4c5d98ef7ff5a32f8cb6" candidate_uuid: "78a82770776349429229de36040aa3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06286426acec47d1955891d8e19ccdd3"
I20240501 13:59:15.922540 29185 raft_consensus.cc:3037] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:15.922650 29315 raft_consensus.cc:3037] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:15.925585 29185 raft_consensus.cc:2454] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a82770776349429229de36040aa3c9 in term 1.
I20240501 13:59:15.925585 29315 raft_consensus.cc:2454] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a82770776349429229de36040aa3c9 in term 1.
I20240501 13:59:15.926050 29380 leader_election.cc:302] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 461ff760d8944577b240697a6eb0b736, 78a82770776349429229de36040aa3c9; no voters:
I20240501 13:59:15.926416 29514 raft_consensus.cc:2781] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:15.926869 29514 raft_consensus.cc:686] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [term 1 LEADER]: Becoming Leader. State: Replica: 78a82770776349429229de36040aa3c9, State: Running, Role: LEADER
I20240501 13:59:15.927448 29514 consensus_queue.cc:237] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } }
I20240501 13:59:15.932235 28783 catalog_manager.cc:5484] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78a82770776349429229de36040aa3c9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "78a82770776349429229de36040aa3c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78a82770776349429229de36040aa3c9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 36767 } health_report { overall_health: HEALTHY } } }
W20240501 13:59:15.949569 29491 tablet.cc:2369] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:15.962742 28740 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
W20240501 13:59:15.963455 29361 tablet.cc:2369] T 62963132761a4c5d98ef7ff5a32f8cb6 P 06286426acec47d1955891d8e19ccdd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:15.965867 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cd0cff7c8bf9440f95e69565a98979fd to finish bootstrapping
I20240501 13:59:15.976723 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 676557eea03049649831d6579fce3378 to finish bootstrapping
I20240501 13:59:15.985808 28740 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 461ff760d8944577b240697a6eb0b736 to finish bootstrapping
I20240501 13:59:15.994578 28740 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06286426acec47d1955891d8e19ccdd3 to finish bootstrapping
W20240501 13:59:15.996532 29231 tablet.cc:2369] T 62963132761a4c5d98ef7ff5a32f8cb6 P 461ff760d8944577b240697a6eb0b736: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:16.002455 28740 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 78a82770776349429229de36040aa3c9 to finish bootstrapping
I20240501 13:59:16.367156 29514 consensus_queue.cc:1035] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06286426acec47d1955891d8e19ccdd3" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:16.388209 29533 consensus_queue.cc:1035] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "461ff760d8944577b240697a6eb0b736" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:16.507617 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:16.519985 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:16.519994 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:16.522624 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:16.542184 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.131846 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.149267 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.149626 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:17.176452 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.178440 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.338027 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.343982 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.355715 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.368165 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.382987 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:17.530941 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.535795 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.544155 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:17.563228 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:17.577591 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:18.109109 28782 ts_manager.cc:292] Set tserver state for cd0cff7c8bf9440f95e69565a98979fd to MAINTENANCE_MODE
I20240501 13:59:19.507665 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.508522 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:19.509299 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.516351 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.525357 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.678673 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.688066 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.695061 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.702068 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:19.708092 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.876734 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.890285 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.915014 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:19.917143 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:19.917564 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.055868 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.064708 29035 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.067476 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.068218 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:20.088416 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.235613 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 28973
I20240501 13:59:20.237038 29095 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:20.333189 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 28973
W20240501 13:59:20.347862 28767 connection.cc:535] server connection from 127.28.17.2:52523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 13:59:20.829278 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.829967 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:20.830310 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.830392 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.962954 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.966265 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.974169 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:20.974165 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:21.140826 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:21.150689 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:21.150887 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:21.158860 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:21.301580 29295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:21.309514 29165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:21.310149 29425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:21.317862 28904 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:21.460508 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 28842
I20240501 13:59:21.461939 28964 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:21.564491 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 28842
I20240501 13:59:21.580519 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 29103
I20240501 13:59:21.581686 29224 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:21.689842 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 29103
I20240501 13:59:21.710525 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 29233
I20240501 13:59:21.711570 29354 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:21.817977 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 29233
W20240501 13:59:21.833400 29381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.17.4:37821 (error 108)
I20240501 13:59:21.833563 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 29363
I20240501 13:59:21.835002 29485 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 13:59:21.835637 29380 consensus_peers.cc:485] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 -> Peer 461ff760d8944577b240697a6eb0b736 (127.28.17.3:42361): Couldn't send request to peer 461ff760d8944577b240697a6eb0b736. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:42361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:21.943161 29381 consensus_peers.cc:485] T 62963132761a4c5d98ef7ff5a32f8cb6 P 78a82770776349429229de36040aa3c9 -> Peer 06286426acec47d1955891d8e19ccdd3 (127.28.17.4:37821): Couldn't send request to peer 06286426acec47d1955891d8e19ccdd3. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.4:37821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:59:21.943274 29380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.3:42361, user_credentials={real_user=slave}} blocked reactor thread for 107611us
W20240501 13:59:21.943392 29381 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.4:37821, user_credentials={real_user=slave}} blocked reactor thread for 108078us
I20240501 13:59:21.943917 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 29363
I20240501 13:59:21.959846 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 28752
I20240501 13:59:21.961040 28813 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:22.090008 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 28752
2024-05-01T13:59:22Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (9492 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (9492 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240501 13:59:22.130178 28740 test_util.cc:268] Using random seed: -457269540
I20240501 13:59:22.131969 28740 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:22.132051 28740 ts_itest-base.cc:113] --------------
I20240501 13:59:22.132088 28740 ts_itest-base.cc:114] 5 tablet servers
I20240501 13:59:22.132153 28740 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:22.132205 28740 ts_itest-base.cc:116] --------------
2024-05-01T13:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:22Z Disabled control of system clock
I20240501 13:59:22.153803 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:38375
--webserver_interface=127.28.17.62
--webserver_port=0
--builtin_ntp_servers=127.28.17.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.17.62:38375
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 13:59:22.483898 29963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:22.484251 29963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:22.484397 29963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:22.491932 29963 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:22.492069 29963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:22.492120 29963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:22.492199 29963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:22.492257 29963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:22.501565 29963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.17.62:38375
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:38375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.17.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:22.502982 29963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:22.504786 29963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:22.516458 29968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:22.516985 29971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:22.517594 29969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:22.518224 29963 server_base.cc:1034] running on GCE node
I20240501 13:59:22.519104 29963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:22.520711 29963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:22.521925 29963 hybrid_clock.cc:648] HybridClock initialized: now 1714571962521877 us; error 42 us; skew 500 ppm
I20240501 13:59:22.522294 29963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:22.524739 29963 webserver.cc:460] Webserver started at http://127.28.17.62:33807/ using document root <none> and password file <none>
I20240501 13:59:22.525506 29963 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:22.525624 29963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:22.525959 29963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:22.528756 29963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b45ad882814d406db314f3663e3af6db"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-dx8c"
I20240501 13:59:22.529605 29963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b45ad882814d406db314f3663e3af6db"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-dx8c"
I20240501 13:59:22.535283 29963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:22.539088 29977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:22.540612 29963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 13:59:22.540802 29963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b45ad882814d406db314f3663e3af6db"
format_stamp: "Formatted at 2024-05-01 13:59:22 on dist-test-slave-dx8c"
I20240501 13:59:22.541007 29963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:22.564905 29963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:22.565824 29963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:22.566099 29963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:22.583953 29963 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.62:38375
I20240501 13:59:22.583974 30028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.62:38375 every 8 connection(s)
I20240501 13:59:22.585433 29963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:22.589361 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:22.591217 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 29963
I20240501 13:59:22.591470 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:22.600417 30029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db: Bootstrap starting.
I20240501 13:59:22.603837 30029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:22.605154 30029 log.cc:824] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:22.607982 30029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db: No bootstrap required, opened a new log
I20240501 13:59:22.614449 30029 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ad882814d406db314f3663e3af6db" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 38375 } }
I20240501 13:59:22.614881 30029 raft_consensus.cc:374] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:22.614984 30029 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45ad882814d406db314f3663e3af6db, State: Initialized, Role: FOLLOWER
I20240501 13:59:22.615641 30029 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ad882814d406db314f3663e3af6db" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 38375 } }
I20240501 13:59:22.615846 30029 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:22.615967 30029 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:22.616102 30029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:22.618134 30029 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ad882814d406db314f3663e3af6db" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 38375 } }
I20240501 13:59:22.618718 30029 leader_election.cc:302] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45ad882814d406db314f3663e3af6db; no voters:
I20240501 13:59:22.619395 30029 leader_election.cc:288] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:22.619550 30034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:22.620518 30034 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [term 1 LEADER]: Becoming Leader. State: Replica: b45ad882814d406db314f3663e3af6db, State: Running, Role: LEADER
I20240501 13:59:22.621032 30034 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ad882814d406db314f3663e3af6db" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 38375 } }
I20240501 13:59:22.621420 30029 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:22.623656 30036 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45ad882814d406db314f3663e3af6db. Latest consensus state: current_term: 1 leader_uuid: "b45ad882814d406db314f3663e3af6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ad882814d406db314f3663e3af6db" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 38375 } } }
I20240501 13:59:22.623987 30036 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:22.624390 30035 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b45ad882814d406db314f3663e3af6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ad882814d406db314f3663e3af6db" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 38375 } } }
I20240501 13:59:22.624656 30035 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:22.625034 30042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:22.631240 30042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:22.641340 30042 catalog_manager.cc:1347] Generated new cluster ID: 7e7315659e9f4088a339c2a2030f9f30
I20240501 13:59:22.641482 30042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:22.660259 30042 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:22.661525 30042 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:22.679553 30042 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b45ad882814d406db314f3663e3af6db: Generated new TSK 0
I20240501 13:59:22.680461 30042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:22.687351 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.1:0
--local_ip_for_outbound_sockets=127.28.17.1
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--builtin_ntp_servers=127.28.17.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:23.018445 30053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:23.018813 30053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:23.018911 30053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:23.026587 30053 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:23.026731 30053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:23.026860 30053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.1
I20240501 13:59:23.036186 30053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.1
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:23.037575 30053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:23.039392 30053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:23.051883 30058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.052449 30061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.053056 30059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:23.053579 30053 server_base.cc:1034] running on GCE node
I20240501 13:59:23.054261 30053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:23.055301 30053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:23.056519 30053 hybrid_clock.cc:648] HybridClock initialized: now 1714571963056472 us; error 40 us; skew 500 ppm
I20240501 13:59:23.056855 30053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:23.059188 30053 webserver.cc:460] Webserver started at http://127.28.17.1:39413/ using document root <none> and password file <none>
I20240501 13:59:23.059916 30053 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:23.060032 30053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:23.060353 30053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:23.063053 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f8c38b87ddfa4e94bea9e4496666a777"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.063831 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f8c38b87ddfa4e94bea9e4496666a777"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.069226 30053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:23.072839 30067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.074383 30053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:23.074616 30053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f8c38b87ddfa4e94bea9e4496666a777"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.074812 30053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:23.095602 30053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:23.096446 30053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:23.096695 30053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:23.097654 30053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:23.099239 30053 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:23.099340 30053 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.099448 30053 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:23.099514 30053 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.128801 30053 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.1:44499
I20240501 13:59:23.128851 30179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.1:44499 every 8 connection(s)
I20240501 13:59:23.130136 30053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:23.135915 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 30053
I20240501 13:59:23.136088 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:23.139364 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.2:0
--local_ip_for_outbound_sockets=127.28.17.2
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--builtin_ntp_servers=127.28.17.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:23.146108 30180 heartbeater.cc:347] Connected to a master server at 127.28.17.62:38375
I20240501 13:59:23.146458 30180 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:23.147347 30180 heartbeater.cc:511] Master 127.28.17.62:38375 requested a full tablet report, sending...
I20240501 13:59:23.149731 29994 ts_manager.cc:194] Registered new tserver with Master: f8c38b87ddfa4e94bea9e4496666a777 (127.28.17.1:44499)
I20240501 13:59:23.151278 29994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.1:34775
W20240501 13:59:23.465724 30184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:23.466040 30184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:23.466178 30184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:23.473657 30184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:23.473811 30184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:23.473968 30184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.2
I20240501 13:59:23.483192 30184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.2
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:23.484535 30184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:23.486367 30184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:23.498293 30189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.498713 30190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.499284 30192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:23.500110 30184 server_base.cc:1034] running on GCE node
I20240501 13:59:23.500756 30184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:23.501696 30184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:23.502880 30184 hybrid_clock.cc:648] HybridClock initialized: now 1714571963502825 us; error 56 us; skew 500 ppm
I20240501 13:59:23.503180 30184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:23.505378 30184 webserver.cc:460] Webserver started at http://127.28.17.2:46863/ using document root <none> and password file <none>
I20240501 13:59:23.506101 30184 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:23.506209 30184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:23.506564 30184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:23.509119 30184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "efa8769c4cb24f3bbcadd68caa1c23be"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.509850 30184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "efa8769c4cb24f3bbcadd68caa1c23be"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.515225 30184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:23.518788 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.520195 30184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:23.520363 30184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "efa8769c4cb24f3bbcadd68caa1c23be"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.520571 30184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:23.543918 30184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:23.544884 30184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:23.545173 30184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:23.546213 30184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:23.547870 30184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:23.547961 30184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.548049 30184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:23.548100 30184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.577593 30184 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.2:39115
I20240501 13:59:23.577643 30310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.2:39115 every 8 connection(s)
I20240501 13:59:23.578985 30184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:23.586412 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 30184
I20240501 13:59:23.586627 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:23.589979 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.3:0
--local_ip_for_outbound_sockets=127.28.17.3
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--builtin_ntp_servers=127.28.17.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:23.593443 30311 heartbeater.cc:347] Connected to a master server at 127.28.17.62:38375
I20240501 13:59:23.593792 30311 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:23.594738 30311 heartbeater.cc:511] Master 127.28.17.62:38375 requested a full tablet report, sending...
I20240501 13:59:23.596132 29994 ts_manager.cc:194] Registered new tserver with Master: efa8769c4cb24f3bbcadd68caa1c23be (127.28.17.2:39115)
I20240501 13:59:23.596899 29994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.2:56849
W20240501 13:59:23.919713 30314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:23.920006 30314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:23.920100 30314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:23.928002 30314 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:23.928177 30314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:23.928316 30314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.3
I20240501 13:59:23.937449 30314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.3
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:23.938853 30314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:23.940569 30314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:23.952903 30319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.953519 30322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:23.954093 30320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:23.954851 30314 server_base.cc:1034] running on GCE node
I20240501 13:59:23.955529 30314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:23.956513 30314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:23.957751 30314 hybrid_clock.cc:648] HybridClock initialized: now 1714571963957699 us; error 44 us; skew 500 ppm
I20240501 13:59:23.958106 30314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:23.960547 30314 webserver.cc:460] Webserver started at http://127.28.17.3:35441/ using document root <none> and password file <none>
I20240501 13:59:23.961328 30314 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:23.961443 30314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:23.961779 30314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:23.964592 30314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6ff7535ac77b4510a13d804af70c87f3"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.965409 30314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6ff7535ac77b4510a13d804af70c87f3"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.971026 30314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 13:59:23.974817 30328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:23.976334 30314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:23.976511 30314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6ff7535ac77b4510a13d804af70c87f3"
format_stamp: "Formatted at 2024-05-01 13:59:23 on dist-test-slave-dx8c"
I20240501 13:59:23.976717 30314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:24.005425 30314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:24.006383 30314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:24.006702 30314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:24.007763 30314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:24.009373 30314 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:24.009474 30314 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.009588 30314 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:24.009653 30314 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.039579 30314 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.3:39925
I20240501 13:59:24.039661 30440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.3:39925 every 8 connection(s)
I20240501 13:59:24.040931 30314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:24.050647 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 30314
I20240501 13:59:24.050828 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:24.054112 30441 heartbeater.cc:347] Connected to a master server at 127.28.17.62:38375
I20240501 13:59:24.054339 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.4:0
--local_ip_for_outbound_sockets=127.28.17.4
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--builtin_ntp_servers=127.28.17.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:24.054481 30441 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:24.055606 30441 heartbeater.cc:511] Master 127.28.17.62:38375 requested a full tablet report, sending...
I20240501 13:59:24.057235 29994 ts_manager.cc:194] Registered new tserver with Master: 6ff7535ac77b4510a13d804af70c87f3 (127.28.17.3:39925)
I20240501 13:59:24.058028 29994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.3:56539
I20240501 13:59:24.154207 30180 heartbeater.cc:503] Master 127.28.17.62:38375 was elected leader, sending a full tablet report...
W20240501 13:59:24.391235 30444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:24.391537 30444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:24.391654 30444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:24.399152 30444 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:24.399279 30444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:24.399426 30444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.4
I20240501 13:59:24.408574 30444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.4
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:24.409941 30444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:24.411826 30444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:24.424764 30452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:24.424975 30449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:24.425334 30450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:24.426069 30444 server_base.cc:1034] running on GCE node
I20240501 13:59:24.426818 30444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:24.427858 30444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:24.429064 30444 hybrid_clock.cc:648] HybridClock initialized: now 1714571964429019 us; error 40 us; skew 500 ppm
I20240501 13:59:24.429400 30444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:24.431757 30444 webserver.cc:460] Webserver started at http://127.28.17.4:36013/ using document root <none> and password file <none>
I20240501 13:59:24.432560 30444 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:24.432667 30444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:24.432992 30444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:24.435758 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1302031b709b4356a246a2491cbcee04"
format_stamp: "Formatted at 2024-05-01 13:59:24 on dist-test-slave-dx8c"
I20240501 13:59:24.436570 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1302031b709b4356a246a2491cbcee04"
format_stamp: "Formatted at 2024-05-01 13:59:24 on dist-test-slave-dx8c"
I20240501 13:59:24.442379 30444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 13:59:24.446107 30458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.447659 30444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:24.447820 30444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1302031b709b4356a246a2491cbcee04"
format_stamp: "Formatted at 2024-05-01 13:59:24 on dist-test-slave-dx8c"
I20240501 13:59:24.448029 30444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:24.475294 30444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:24.476215 30444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:24.476495 30444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:24.477515 30444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:24.479225 30444 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:24.479333 30444 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.479439 30444 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:24.479506 30444 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.510962 30444 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.4:35881
I20240501 13:59:24.511021 30570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.4:35881 every 8 connection(s)
I20240501 13:59:24.512334 30444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:24.522786 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 30444
I20240501 13:59:24.523012 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:24.525879 30571 heartbeater.cc:347] Connected to a master server at 127.28.17.62:38375
I20240501 13:59:24.526270 30571 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:24.526824 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:0
--local_ip_for_outbound_sockets=127.28.17.5
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--builtin_ntp_servers=127.28.17.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:24.527468 30571 heartbeater.cc:511] Master 127.28.17.62:38375 requested a full tablet report, sending...
I20240501 13:59:24.529227 29994 ts_manager.cc:194] Registered new tserver with Master: 1302031b709b4356a246a2491cbcee04 (127.28.17.4:35881)
I20240501 13:59:24.530138 29994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.4:47839
I20240501 13:59:24.599566 30311 heartbeater.cc:503] Master 127.28.17.62:38375 was elected leader, sending a full tablet report...
W20240501 13:59:24.874934 30574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:24.875260 30574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:24.875397 30574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:24.883412 30574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:24.883570 30574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:24.883724 30574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 13:59:24.893033 30574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:38375
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:24.894419 30574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:24.896266 30574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:24.908821 30582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:24.909317 30579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:24.910107 30580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:24.910602 30574 server_base.cc:1034] running on GCE node
I20240501 13:59:24.911249 30574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:24.912397 30574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:24.913633 30574 hybrid_clock.cc:648] HybridClock initialized: now 1714571964913561 us; error 65 us; skew 500 ppm
I20240501 13:59:24.914180 30574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:24.916910 30574 webserver.cc:460] Webserver started at http://127.28.17.5:38431/ using document root <none> and password file <none>
I20240501 13:59:24.917711 30574 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:24.917814 30574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:24.918099 30574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:24.920841 30574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e20d1440bbc6470d89a472feb78f7f4c"
format_stamp: "Formatted at 2024-05-01 13:59:24 on dist-test-slave-dx8c"
I20240501 13:59:24.921658 30574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e20d1440bbc6470d89a472feb78f7f4c"
format_stamp: "Formatted at 2024-05-01 13:59:24 on dist-test-slave-dx8c"
I20240501 13:59:24.927582 30574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240501 13:59:24.931393 30588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.932976 30574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:24.933136 30574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e20d1440bbc6470d89a472feb78f7f4c"
format_stamp: "Formatted at 2024-05-01 13:59:24 on dist-test-slave-dx8c"
I20240501 13:59:24.933353 30574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:24.953197 30574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:24.954114 30574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:24.954389 30574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:24.955408 30574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:24.956965 30574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:24.957052 30574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.957155 30574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:24.957216 30574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:24.987994 30574 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:34187
I20240501 13:59:24.988045 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:34187 every 8 connection(s)
I20240501 13:59:24.989573 30574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:24.995849 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 30574
I20240501 13:59:24.996023 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:25.003057 30701 heartbeater.cc:347] Connected to a master server at 127.28.17.62:38375
I20240501 13:59:25.003401 30701 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:25.004297 30701 heartbeater.cc:511] Master 127.28.17.62:38375 requested a full tablet report, sending...
I20240501 13:59:25.005896 29994 ts_manager.cc:194] Registered new tserver with Master: e20d1440bbc6470d89a472feb78f7f4c (127.28.17.5:34187)
I20240501 13:59:25.006772 29994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:35855
I20240501 13:59:25.012161 28740 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:25.029042 29994 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49248:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:25.060775 30441 heartbeater.cc:503] Master 127.28.17.62:38375 was elected leader, sending a full tablet report...
I20240501 13:59:25.074692 30246 tablet_service.cc:1467] Processing CreateTablet for tablet c52d317f2675493a893c8be1daf7d960 (DEFAULT_TABLE table=TestTable [id=5063f93672c84680b46f3b5b033b5957]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.075848 30506 tablet_service.cc:1467] Processing CreateTablet for tablet c52d317f2675493a893c8be1daf7d960 (DEFAULT_TABLE table=TestTable [id=5063f93672c84680b46f3b5b033b5957]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.076437 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52d317f2675493a893c8be1daf7d960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.077013 30506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52d317f2675493a893c8be1daf7d960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.077168 30376 tablet_service.cc:1467] Processing CreateTablet for tablet c52d317f2675493a893c8be1daf7d960 (DEFAULT_TABLE table=TestTable [id=5063f93672c84680b46f3b5b033b5957]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:25.078712 30376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52d317f2675493a893c8be1daf7d960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:25.088450 30719 tablet_bootstrap.cc:492] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04: Bootstrap starting.
I20240501 13:59:25.090844 30720 tablet_bootstrap.cc:492] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be: Bootstrap starting.
I20240501 13:59:25.092293 30719 tablet_bootstrap.cc:654] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.092756 30721 tablet_bootstrap.cc:492] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3: Bootstrap starting.
I20240501 13:59:25.094204 30719 log.cc:824] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:25.094601 30720 tablet_bootstrap.cc:654] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.095881 30720 log.cc:824] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:25.096817 30721 tablet_bootstrap.cc:654] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:25.097812 30719 tablet_bootstrap.cc:492] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04: No bootstrap required, opened a new log
I20240501 13:59:25.097967 30721 log.cc:824] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:25.098073 30719 ts_tablet_manager.cc:1367] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 13:59:25.098995 30720 tablet_bootstrap.cc:492] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be: No bootstrap required, opened a new log
I20240501 13:59:25.099275 30720 ts_tablet_manager.cc:1367] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 13:59:25.100839 30721 tablet_bootstrap.cc:492] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3: No bootstrap required, opened a new log
I20240501 13:59:25.101116 30721 ts_tablet_manager.cc:1367] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 13:59:25.105134 30719 raft_consensus.cc:348] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.105530 30719 raft_consensus.cc:374] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.105660 30719 raft_consensus.cc:729] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1302031b709b4356a246a2491cbcee04, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.106307 30719 consensus_queue.cc:260] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.107452 30571 heartbeater.cc:503] Master 127.28.17.62:38375 was elected leader, sending a full tablet report...
I20240501 13:59:25.107937 30719 ts_tablet_manager.cc:1398] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20240501 13:59:25.108098 30720 raft_consensus.cc:348] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.108542 30720 raft_consensus.cc:374] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.108660 30720 raft_consensus.cc:729] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa8769c4cb24f3bbcadd68caa1c23be, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.109299 30720 consensus_queue.cc:260] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.110577 30721 raft_consensus.cc:348] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.111059 30721 raft_consensus.cc:374] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:25.111107 30720 ts_tablet_manager.cc:1398] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20240501 13:59:25.111189 30721 raft_consensus.cc:729] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff7535ac77b4510a13d804af70c87f3, State: Initialized, Role: FOLLOWER
I20240501 13:59:25.112300 30721 consensus_queue.cc:260] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.113951 30721 ts_tablet_manager.cc:1398] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20240501 13:59:25.264513 30572 tablet.cc:2369] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:25.285161 30726 raft_consensus.cc:482] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:25.285466 30726 raft_consensus.cc:504] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.287273 30726 leader_election.cc:288] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1302031b709b4356a246a2491cbcee04 (127.28.17.4:35881), 6ff7535ac77b4510a13d804af70c87f3 (127.28.17.3:39925)
W20240501 13:59:25.293351 30442 tablet.cc:2369] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:25.295456 30396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52d317f2675493a893c8be1daf7d960" candidate_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff7535ac77b4510a13d804af70c87f3" is_pre_election: true
I20240501 13:59:25.295459 30526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52d317f2675493a893c8be1daf7d960" candidate_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1302031b709b4356a246a2491cbcee04" is_pre_election: true
I20240501 13:59:25.295902 30396 raft_consensus.cc:2454] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa8769c4cb24f3bbcadd68caa1c23be in term 0.
I20240501 13:59:25.295898 30526 raft_consensus.cc:2454] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa8769c4cb24f3bbcadd68caa1c23be in term 0.
I20240501 13:59:25.296484 30199 leader_election.cc:302] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff7535ac77b4510a13d804af70c87f3, efa8769c4cb24f3bbcadd68caa1c23be; no voters:
I20240501 13:59:25.296854 30726 raft_consensus.cc:2781] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:25.297027 30726 raft_consensus.cc:482] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:25.297139 30726 raft_consensus.cc:3037] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.299206 30726 raft_consensus.cc:504] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.299883 30726 leader_election.cc:288] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [CANDIDATE]: Term 1 election: Requested vote from peers 1302031b709b4356a246a2491cbcee04 (127.28.17.4:35881), 6ff7535ac77b4510a13d804af70c87f3 (127.28.17.3:39925)
I20240501 13:59:25.300421 30396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52d317f2675493a893c8be1daf7d960" candidate_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff7535ac77b4510a13d804af70c87f3"
I20240501 13:59:25.300433 30526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52d317f2675493a893c8be1daf7d960" candidate_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1302031b709b4356a246a2491cbcee04"
I20240501 13:59:25.300630 30396 raft_consensus.cc:3037] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.300628 30526 raft_consensus.cc:3037] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:25.302778 30396 raft_consensus.cc:2454] T c52d317f2675493a893c8be1daf7d960 P 6ff7535ac77b4510a13d804af70c87f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa8769c4cb24f3bbcadd68caa1c23be in term 1.
I20240501 13:59:25.303172 30199 leader_election.cc:302] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff7535ac77b4510a13d804af70c87f3, efa8769c4cb24f3bbcadd68caa1c23be; no voters:
I20240501 13:59:25.303341 30526 raft_consensus.cc:2454] T c52d317f2675493a893c8be1daf7d960 P 1302031b709b4356a246a2491cbcee04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa8769c4cb24f3bbcadd68caa1c23be in term 1.
I20240501 13:59:25.303483 30726 raft_consensus.cc:2781] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:25.304136 30726 raft_consensus.cc:686] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [term 1 LEADER]: Becoming Leader. State: Replica: efa8769c4cb24f3bbcadd68caa1c23be, State: Running, Role: LEADER
I20240501 13:59:25.304597 30726 consensus_queue.cc:237] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } }
I20240501 13:59:25.309253 29993 catalog_manager.cc:5484] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be reported cstate change: term changed from 0 to 1, leader changed from <none> to efa8769c4cb24f3bbcadd68caa1c23be (127.28.17.2). New cstate: current_term: 1 leader_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa8769c4cb24f3bbcadd68caa1c23be" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 39115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:25.331737 30312 tablet.cc:2369] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:25.373060 28740 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 13:59:25.375684 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f8c38b87ddfa4e94bea9e4496666a777 to finish bootstrapping
I20240501 13:59:25.385840 28740 ts_itest-base.cc:233] Waiting for 1 tablets on tserver efa8769c4cb24f3bbcadd68caa1c23be to finish bootstrapping
I20240501 13:59:25.393568 28740 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6ff7535ac77b4510a13d804af70c87f3 to finish bootstrapping
I20240501 13:59:25.401345 28740 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1302031b709b4356a246a2491cbcee04 to finish bootstrapping
I20240501 13:59:25.409229 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e20d1440bbc6470d89a472feb78f7f4c to finish bootstrapping
I20240501 13:59:25.418583 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 30053
I20240501 13:59:25.419658 30175 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:25.500036 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 30053
I20240501 13:59:25.710264 30726 consensus_queue.cc:1035] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1302031b709b4356a246a2491cbcee04" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 35881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:25.721694 30741 consensus_queue.cc:1035] T c52d317f2675493a893c8be1daf7d960 P efa8769c4cb24f3bbcadd68caa1c23be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff7535ac77b4510a13d804af70c87f3" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 39925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:25.982542 30506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:25.987145 30636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:25.992336 30246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:25.993166 30376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.009599 30701 heartbeater.cc:503] Master 127.28.17.62:38375 was elected leader, sending a full tablet report...
I20240501 13:59:26.125842 30506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.131325 30376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.133042 30636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.140136 30246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:26.306023 30506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.310101 30376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.317845 30636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.318964 30246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:26.943924 30636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.943957 30506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.951529 30376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:26.951741 30246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:27.080778 30506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:27.083079 30376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:27.086230 30636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:27.090639 30246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:27.274694 30506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:27.294165 30376 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:27.296135 30636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:27.297264 30246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:27.445518 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 30184
I20240501 13:59:27.447046 30306 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:27.543689 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 30184
W20240501 13:59:27.558085 30461 connection.cc:535] server connection from 127.28.17.2:50871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 13:59:27.558393 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 30314
I20240501 13:59:27.559402 30436 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:27.653801 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 30314
I20240501 13:59:27.668067 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 30444
I20240501 13:59:27.669194 30565 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:27.764662 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 30444
W20240501 13:59:27.778867 29978 connection.cc:535] server connection from 127.28.17.4:47839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 13:59:27.779086 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 30574
I20240501 13:59:27.780148 30695 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:27.870564 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 30574
I20240501 13:59:27.883834 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 29963
I20240501 13:59:27.885033 30024 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:27.999884 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 29963
2024-05-01T13:59:28Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (5907 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5907 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240501 13:59:28.038028 28740 test_util.cc:268] Using random seed: -451361682
I20240501 13:59:28.039903 28740 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:28.039968 28740 ts_itest-base.cc:113] --------------
I20240501 13:59:28.040015 28740 ts_itest-base.cc:114] 7 tablet servers
I20240501 13:59:28.040063 28740 ts_itest-base.cc:115] 1 replicas per TS
I20240501 13:59:28.040120 28740 ts_itest-base.cc:116] --------------
2024-05-01T13:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:28Z Disabled control of system clock
I20240501 13:59:28.063623 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:46735
--webserver_interface=127.28.17.62
--webserver_port=0
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.17.62:46735
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 13:59:28.386584 30942 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 13:59:28.386946 30942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:28.387046 30942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:28.387143 30942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:28.394731 30942 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:28.394860 30942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:28.394924 30942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:28.394987 30942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:28.395053 30942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:28.404840 30942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.17.62:46735
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.17.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:28.406230 30942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:28.408152 30942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:28.420627 30950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.421300 30948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.421993 30947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:28.422300 30942 server_base.cc:1034] running on GCE node
I20240501 13:59:28.423326 30942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:28.425021 30942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:28.426306 30942 hybrid_clock.cc:648] HybridClock initialized: now 1714571968426232 us; error 70 us; skew 500 ppm
I20240501 13:59:28.426738 30942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:28.433637 30942 webserver.cc:460] Webserver started at http://127.28.17.62:35207/ using document root <none> and password file <none>
I20240501 13:59:28.434459 30942 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:28.434654 30942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:28.434983 30942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:28.437780 30942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "26e34a0aa8454cc9810f40277fc27268"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-dx8c"
I20240501 13:59:28.438673 30942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "26e34a0aa8454cc9810f40277fc27268"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-dx8c"
I20240501 13:59:28.444473 30942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:28.448504 30956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.450163 30942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:28.450388 30942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
uuid: "26e34a0aa8454cc9810f40277fc27268"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-dx8c"
I20240501 13:59:28.450668 30942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:28.464416 30942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:28.465343 30942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:28.465655 30942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:28.483493 30942 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.62:46735
I20240501 13:59:28.483511 31007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.62:46735 every 8 connection(s)
I20240501 13:59:28.484951 30942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:28.489027 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:28.490662 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 30942
I20240501 13:59:28.490852 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:28.499806 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268: Bootstrap starting.
I20240501 13:59:28.503185 31008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:28.504575 31008 log.cc:824] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:28.507459 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268: No bootstrap required, opened a new log
I20240501 13:59:28.513671 31008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e34a0aa8454cc9810f40277fc27268" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 46735 } }
I20240501 13:59:28.514066 31008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:28.514190 31008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e34a0aa8454cc9810f40277fc27268, State: Initialized, Role: FOLLOWER
I20240501 13:59:28.514894 31008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e34a0aa8454cc9810f40277fc27268" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 46735 } }
I20240501 13:59:28.515110 31008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:28.515229 31008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:28.515388 31008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:28.517480 31008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e34a0aa8454cc9810f40277fc27268" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 46735 } }
I20240501 13:59:28.518007 31008 leader_election.cc:302] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26e34a0aa8454cc9810f40277fc27268; no voters:
I20240501 13:59:28.518692 31008 leader_election.cc:288] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:28.518824 31013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:28.519399 31013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [term 1 LEADER]: Becoming Leader. State: Replica: 26e34a0aa8454cc9810f40277fc27268, State: Running, Role: LEADER
I20240501 13:59:28.520040 31013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e34a0aa8454cc9810f40277fc27268" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 46735 } }
I20240501 13:59:28.520999 31008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:28.523005 31014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e34a0aa8454cc9810f40277fc27268. Latest consensus state: current_term: 1 leader_uuid: "26e34a0aa8454cc9810f40277fc27268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e34a0aa8454cc9810f40277fc27268" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 46735 } } }
I20240501 13:59:28.523509 31014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:28.523844 31015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26e34a0aa8454cc9810f40277fc27268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e34a0aa8454cc9810f40277fc27268" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 46735 } } }
I20240501 13:59:28.524129 31015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:28.525205 31021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:28.531194 31021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:28.539621 31021 catalog_manager.cc:1347] Generated new cluster ID: cfc1a958ce444f87aad403cbc6975bf2
I20240501 13:59:28.539757 31021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:28.553901 31021 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:28.555766 31021 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:28.570921 31021 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 26e34a0aa8454cc9810f40277fc27268: Generated new TSK 0
I20240501 13:59:28.571946 31021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:28.587929 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.1:0
--local_ip_for_outbound_sockets=127.28.17.1
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:28.936437 31032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:28.936725 31032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:28.936820 31032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:28.944419 31032 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:28.944553 31032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:28.944671 31032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.1
I20240501 13:59:28.953927 31032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.1
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:28.955313 31032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:28.957026 31032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:28.968871 31038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.969231 31040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.969877 31037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:28.970429 31032 server_base.cc:1034] running on GCE node
I20240501 13:59:28.971029 31032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:28.972007 31032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:28.973203 31032 hybrid_clock.cc:648] HybridClock initialized: now 1714571968973141 us; error 57 us; skew 500 ppm
I20240501 13:59:28.973518 31032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:28.975973 31032 webserver.cc:460] Webserver started at http://127.28.17.1:38349/ using document root <none> and password file <none>
I20240501 13:59:28.976815 31032 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:28.976922 31032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:28.977362 31032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:28.980054 31032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f3c5452a71ac4778977f154db2cb1659"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-dx8c"
I20240501 13:59:28.980825 31032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f3c5452a71ac4778977f154db2cb1659"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-dx8c"
I20240501 13:59:28.986047 31032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 13:59:28.989607 31046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:28.991166 31032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:28.991323 31032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f3c5452a71ac4778977f154db2cb1659"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-dx8c"
I20240501 13:59:28.991533 31032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.011379 31032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.012249 31032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:29.012517 31032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.013478 31032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:29.015012 31032 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:29.015100 31032 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.015205 31032 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:29.015277 31032 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.044432 31032 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.1:44235
I20240501 13:59:29.044497 31158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.1:44235 every 8 connection(s)
I20240501 13:59:29.045754 31032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:29.047618 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 31032
I20240501 13:59:29.047806 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:29.051447 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.2:0
--local_ip_for_outbound_sockets=127.28.17.2
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:29.059840 31159 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:29.060150 31159 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:29.060998 31159 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:29.063225 30973 ts_manager.cc:194] Registered new tserver with Master: f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:29.064802 30973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.1:39891
W20240501 13:59:29.387286 31163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:29.387609 31163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:29.387734 31163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:29.395357 31163 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:29.395515 31163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:29.395661 31163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.2
I20240501 13:59:29.405118 31163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.2
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:29.406538 31163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:29.408641 31163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:29.422029 31168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.422819 31169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.423723 31171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.424360 31163 server_base.cc:1034] running on GCE node
I20240501 13:59:29.425045 31163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:29.426191 31163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:29.427443 31163 hybrid_clock.cc:648] HybridClock initialized: now 1714571969427352 us; error 78 us; skew 500 ppm
I20240501 13:59:29.427909 31163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.430423 31163 webserver.cc:460] Webserver started at http://127.28.17.2:33233/ using document root <none> and password file <none>
I20240501 13:59:29.431290 31163 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.431401 31163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.431747 31163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:29.434556 31163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-dx8c"
I20240501 13:59:29.435434 31163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-dx8c"
I20240501 13:59:29.441278 31163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:29.445163 31177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.446723 31163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:29.446894 31163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-dx8c"
I20240501 13:59:29.447100 31163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.465901 31163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.466822 31163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:29.467087 31163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.468039 31163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:29.469617 31163 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:29.469714 31163 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.469836 31163 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:29.469900 31163 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.502715 31163 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.2:36045
I20240501 13:59:29.502761 31289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.2:36045 every 8 connection(s)
I20240501 13:59:29.504042 31163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:29.510676 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 31163
I20240501 13:59:29.510861 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:29.514094 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.3:0
--local_ip_for_outbound_sockets=127.28.17.3
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:29.517462 31290 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:29.517789 31290 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:29.518689 31290 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:29.520176 30973 ts_manager.cc:194] Registered new tserver with Master: 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:29.521203 30973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.2:42965
W20240501 13:59:29.843525 31293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:29.843822 31293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:29.843919 31293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:29.851440 31293 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:29.851580 31293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:29.851723 31293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.3
I20240501 13:59:29.861016 31293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.3
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:29.863052 31293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:29.865659 31293 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:29.879139 31298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.879719 31301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.880101 31299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.880934 31293 server_base.cc:1034] running on GCE node
I20240501 13:59:29.881737 31293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:29.882797 31293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:29.884009 31293 hybrid_clock.cc:648] HybridClock initialized: now 1714571969883963 us; error 47 us; skew 500 ppm
I20240501 13:59:29.884335 31293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.886664 31293 webserver.cc:460] Webserver started at http://127.28.17.3:35929/ using document root <none> and password file <none>
I20240501 13:59:29.887427 31293 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.887540 31293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.887856 31293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:29.890537 31293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3fc17cbb4edf4c11b1d6132ae4896997"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-dx8c"
I20240501 13:59:29.891355 31293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3fc17cbb4edf4c11b1d6132ae4896997"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-dx8c"
I20240501 13:59:29.896968 31293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:29.900660 31307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.902235 31293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:29.902424 31293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3fc17cbb4edf4c11b1d6132ae4896997"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-dx8c"
I20240501 13:59:29.902727 31293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.928942 31293 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.929865 31293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:29.930150 31293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.931192 31293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:29.932790 31293 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:29.932879 31293 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.932991 31293 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:29.933058 31293 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.964160 31293 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.3:37467
I20240501 13:59:29.964207 31419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.3:37467 every 8 connection(s)
I20240501 13:59:29.965482 31293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:29.973201 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 31293
I20240501 13:59:29.973409 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:29.976626 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.4:0
--local_ip_for_outbound_sockets=127.28.17.4
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:29.978523 31420 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:29.978847 31420 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:29.979698 31420 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:29.981248 30973 ts_manager.cc:194] Registered new tserver with Master: 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3:37467)
I20240501 13:59:29.982303 30973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.3:55281
I20240501 13:59:30.067957 31159 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
W20240501 13:59:30.303948 31423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:30.304250 31423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:30.304353 31423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:30.311791 31423 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:30.311923 31423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:30.312036 31423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.4
I20240501 13:59:30.321058 31423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.4
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:30.322443 31423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:30.324298 31423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:30.336889 31429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.337451 31428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.337931 31431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:30.338694 31423 server_base.cc:1034] running on GCE node
I20240501 13:59:30.339390 31423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:30.340356 31423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:30.341574 31423 hybrid_clock.cc:648] HybridClock initialized: now 1714571970341530 us; error 47 us; skew 500 ppm
I20240501 13:59:30.341886 31423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:30.344254 31423 webserver.cc:460] Webserver started at http://127.28.17.4:39511/ using document root <none> and password file <none>
I20240501 13:59:30.345007 31423 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:30.345124 31423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:30.345438 31423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:30.348116 31423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d43e3c8c63554319b7bd3f1881889c32"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:30.348917 31423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d43e3c8c63554319b7bd3f1881889c32"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:30.354590 31423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:30.358289 31437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.359764 31423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:30.359942 31423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d43e3c8c63554319b7bd3f1881889c32"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:30.360144 31423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:30.400173 31423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:30.401072 31423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:30.401353 31423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:30.402318 31423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:30.403874 31423 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:30.403973 31423 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.404078 31423 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:30.404146 31423 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.433559 31423 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.4:34759
I20240501 13:59:30.433629 31549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.4:34759 every 8 connection(s)
I20240501 13:59:30.434908 31423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:30.435500 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 31423
I20240501 13:59:30.435657 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:30.439145 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:0
--local_ip_for_outbound_sockets=127.28.17.5
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:30.447990 31550 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:30.448369 31550 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:30.449493 31550 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:30.451359 30973 ts_manager.cc:194] Registered new tserver with Master: d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4:34759)
I20240501 13:59:30.452186 30973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.4:37677
I20240501 13:59:30.523945 31290 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
W20240501 13:59:30.771783 31553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:30.772104 31553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:30.772234 31553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:30.779688 31553 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:30.779832 31553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:30.779994 31553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 13:59:30.789114 31553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:30.790454 31553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:30.792191 31553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:30.804097 31558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.804378 31559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:30.805054 31561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:30.805629 31553 server_base.cc:1034] running on GCE node
I20240501 13:59:30.806221 31553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:30.807273 31553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:30.808477 31553 hybrid_clock.cc:648] HybridClock initialized: now 1714571970808426 us; error 42 us; skew 500 ppm
I20240501 13:59:30.808804 31553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:30.811074 31553 webserver.cc:460] Webserver started at http://127.28.17.5:41955/ using document root <none> and password file <none>
I20240501 13:59:30.811828 31553 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:30.811934 31553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:30.812234 31553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:30.814913 31553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b6318df9141f4124938a82e74ce5fdcd"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:30.815678 31553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b6318df9141f4124938a82e74ce5fdcd"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:30.821046 31553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 13:59:30.824710 31567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.826181 31553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:30.826349 31553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b6318df9141f4124938a82e74ce5fdcd"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:30.826594 31553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:30.860394 31553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:30.861387 31553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:30.861686 31553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:30.862768 31553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:30.864341 31553 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:30.864430 31553 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.864545 31553 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:30.864610 31553 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:30.894665 31553 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:46587
I20240501 13:59:30.894711 31679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:46587 every 8 connection(s)
I20240501 13:59:30.896054 31553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:30.897444 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 31553
I20240501 13:59:30.897631 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:30.901389 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.6:0
--local_ip_for_outbound_sockets=127.28.17.6
--webserver_interface=127.28.17.6
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:30.910027 31680 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:30.910387 31680 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:30.911481 31680 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:30.913143 30973 ts_manager.cc:194] Registered new tserver with Master: b6318df9141f4124938a82e74ce5fdcd (127.28.17.5:46587)
I20240501 13:59:30.913909 30973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:55367
I20240501 13:59:30.985493 31420 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
W20240501 13:59:31.239749 31683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:31.240046 31683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:31.240164 31683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:31.247699 31683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:31.247833 31683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:31.247979 31683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.6
I20240501 13:59:31.257154 31683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.17.6
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.6
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:31.258569 31683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:31.260494 31683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:31.276257 31689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:31.276825 31688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:31.277235 31691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:31.277948 31683 server_base.cc:1034] running on GCE node
I20240501 13:59:31.278640 31683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:31.279687 31683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:31.280916 31683 hybrid_clock.cc:648] HybridClock initialized: now 1714571971280869 us; error 40 us; skew 500 ppm
I20240501 13:59:31.281280 31683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:31.283643 31683 webserver.cc:460] Webserver started at http://127.28.17.6:42707/ using document root <none> and password file <none>
I20240501 13:59:31.284412 31683 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:31.284513 31683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:31.284844 31683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:31.287612 31683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9857990f442f4e7f98122344ebe75ffb"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:31.288432 31683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9857990f442f4e7f98122344ebe75ffb"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:31.293891 31683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 13:59:31.297636 31697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:31.299201 31683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:31.299381 31683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9857990f442f4e7f98122344ebe75ffb"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:31.299594 31683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:31.336308 31683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:31.337198 31683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:31.337486 31683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:31.338532 31683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:31.340036 31683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:31.340128 31683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:31.340229 31683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:31.340301 31683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:31.370398 31683 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.6:42705
I20240501 13:59:31.370443 31809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.6:42705 every 8 connection(s)
I20240501 13:59:31.371778 31683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:31.381454 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 31683
I20240501 13:59:31.381632 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 13:59:31.385208 31810 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:31.385311 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.7:0
--local_ip_for_outbound_sockets=127.28.17.7
--webserver_interface=127.28.17.7
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:31.385571 31810 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:31.386456 31810 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:31.388073 30973 ts_manager.cc:194] Registered new tserver with Master: 9857990f442f4e7f98122344ebe75ffb (127.28.17.6:42705)
I20240501 13:59:31.388859 30973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.6:52597
I20240501 13:59:31.455627 31550 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
W20240501 13:59:31.720961 31813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:31.721262 31813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:31.721371 31813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:31.729049 31813 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:31.729195 31813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:31.729342 31813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.7
I20240501 13:59:31.738668 31813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.17.7
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.7
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:31.740007 31813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:31.741747 31813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:31.754096 31818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:31.754649 31821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:31.755162 31819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:31.755923 31813 server_base.cc:1034] running on GCE node
I20240501 13:59:31.756640 31813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:31.757620 31813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:31.758811 31813 hybrid_clock.cc:648] HybridClock initialized: now 1714571971758756 us; error 49 us; skew 500 ppm
I20240501 13:59:31.759138 31813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:31.761418 31813 webserver.cc:460] Webserver started at http://127.28.17.7:33937/ using document root <none> and password file <none>
I20240501 13:59:31.762164 31813 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:31.762271 31813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:31.762676 31813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:31.765393 31813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "92a0ab1d123c4dcba63e5e5064e059f6"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:31.766196 31813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "92a0ab1d123c4dcba63e5e5064e059f6"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:31.771524 31813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 13:59:31.775022 31827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:31.776418 31813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 13:59:31.776585 31813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "92a0ab1d123c4dcba63e5e5064e059f6"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:31.776789 31813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:31.790747 31813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:31.791610 31813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:31.791878 31813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:31.792826 31813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:31.794291 31813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:31.794385 31813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:31.794492 31813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:31.794584 31813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240501 13:59:31.824060 31813 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.7:45721
I20240501 13:59:31.824103 31939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.7:45721 every 8 connection(s)
I20240501 13:59:31.825389 31813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:31.832504 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 31813
I20240501 13:59:31.832721 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 13:59:31.838482 31940 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:31.838812 31940 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:31.839695 31940 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:31.841140 30973 ts_manager.cc:194] Registered new tserver with Master: 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7:45721)
I20240501 13:59:31.841912 30973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.7:41151
I20240501 13:59:31.848897 28740 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 13:59:31.864665 30973 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43758:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 13:59:31.892233 31485 tablet_service.cc:1467] Processing CreateTablet for tablet 81f2b5656f784e8caa4ade4c76228899 (DEFAULT_TABLE table=TestTable [id=9e3a5ed96a7647e99c56110ed9fbfd8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:31.893558 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81f2b5656f784e8caa4ade4c76228899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:31.904013 31954 tablet_bootstrap.cc:492] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Bootstrap starting.
I20240501 13:59:31.907243 31954 tablet_bootstrap.cc:654] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:31.908305 31954 log.cc:824] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:31.910727 31954 tablet_bootstrap.cc:492] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: No bootstrap required, opened a new log
I20240501 13:59:31.910967 31954 ts_tablet_manager.cc:1367] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 13:59:31.917129 31680 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
I20240501 13:59:31.916977 31954 raft_consensus.cc:348] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:31.917341 31954 raft_consensus.cc:374] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:31.917438 31954 raft_consensus.cc:729] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43e3c8c63554319b7bd3f1881889c32, State: Initialized, Role: FOLLOWER
I20240501 13:59:31.918133 31954 consensus_queue.cc:260] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:31.918434 31954 raft_consensus.cc:388] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:31.918589 31954 raft_consensus.cc:482] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:31.918767 31954 raft_consensus.cc:3037] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:31.920755 31954 raft_consensus.cc:504] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:31.921284 31954 leader_election.cc:302] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43e3c8c63554319b7bd3f1881889c32; no voters:
I20240501 13:59:31.921967 31954 leader_election.cc:288] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:31.922128 31956 raft_consensus.cc:2781] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:31.925415 31956 raft_consensus.cc:686] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 LEADER]: Becoming Leader. State: Replica: d43e3c8c63554319b7bd3f1881889c32, State: Running, Role: LEADER
I20240501 13:59:31.926146 31956 consensus_queue.cc:237] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:31.926337 31954 ts_tablet_manager.cc:1398] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240501 13:59:31.931739 30973 catalog_manager.cc:5484] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4). New cstate: current_term: 1 leader_uuid: "d43e3c8c63554319b7bd3f1881889c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:31.952613 28740 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 13:59:31.954798 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f3c5452a71ac4778977f154db2cb1659 to finish bootstrapping
I20240501 13:59:31.964007 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 73ddc7b55a734843abb88e9f6d2be4b7 to finish bootstrapping
I20240501 13:59:31.973147 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3fc17cbb4edf4c11b1d6132ae4896997 to finish bootstrapping
I20240501 13:59:31.981581 28740 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d43e3c8c63554319b7bd3f1881889c32 to finish bootstrapping
I20240501 13:59:31.989600 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b6318df9141f4124938a82e74ce5fdcd to finish bootstrapping
I20240501 13:59:31.998553 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9857990f442f4e7f98122344ebe75ffb to finish bootstrapping
I20240501 13:59:32.007789 28740 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 92a0ab1d123c4dcba63e5e5064e059f6 to finish bootstrapping
I20240501 13:59:32.017127 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 31293
I20240501 13:59:32.018164 31415 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:32.099097 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 31293
I20240501 13:59:32.112550 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 31423
I20240501 13:59:32.113662 31545 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:32.199002 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 31423
I20240501 13:59:32.212199 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 31553
I20240501 13:59:32.213279 31675 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:32.293119 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 31553
I20240501 13:59:32.305830 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 31683
I20240501 13:59:32.306988 31805 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:32.387485 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 31683
I20240501 13:59:32.400374 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 31813
I20240501 13:59:32.401490 31935 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:32.481294 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 31813
I20240501 13:59:36.245487 30973 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43758:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:36.265512 31093 tablet_service.cc:1467] Processing CreateTablet for tablet 4740a81166714d6bb90c11a657f0a8d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbf4be62827b4639bb92e73b579e499a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:36.265554 31225 tablet_service.cc:1467] Processing CreateTablet for tablet 77be73948b1248fa8e6dd0de4dd65467 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbf4be62827b4639bb92e73b579e499a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:36.265534 31094 tablet_service.cc:1467] Processing CreateTablet for tablet 6b71845f2e464353806c6306e4799456 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fbf4be62827b4639bb92e73b579e499a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:36.266932 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4740a81166714d6bb90c11a657f0a8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.266961 31225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77be73948b1248fa8e6dd0de4dd65467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.267361 31094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b71845f2e464353806c6306e4799456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.278721 31981 tablet_bootstrap.cc:492] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:36.278867 31982 tablet_bootstrap.cc:492] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Bootstrap starting.
I20240501 13:59:36.282488 31981 tablet_bootstrap.cc:654] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:36.282488 31982 tablet_bootstrap.cc:654] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:36.283911 31982 log.cc:824] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:36.283911 31981 log.cc:824] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:36.287029 31981 tablet_bootstrap.cc:492] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:36.287282 31982 tablet_bootstrap.cc:492] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: No bootstrap required, opened a new log
I20240501 13:59:36.287420 31981 ts_tablet_manager.cc:1367] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 13:59:36.287577 31982 ts_tablet_manager.cc:1367] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240501 13:59:36.296483 31981 raft_consensus.cc:348] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.296483 31982 raft_consensus.cc:348] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.296957 31982 raft_consensus.cc:374] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:36.296962 31981 raft_consensus.cc:374] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:36.297071 31982 raft_consensus.cc:729] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Initialized, Role: FOLLOWER
I20240501 13:59:36.297071 31981 raft_consensus.cc:729] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:36.297847 31982 consensus_queue.cc:260] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.297845 31981 consensus_queue.cc:260] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.298105 31982 raft_consensus.cc:388] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:36.298105 31981 raft_consensus.cc:388] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:36.298242 31982 raft_consensus.cc:482] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:36.298242 31981 raft_consensus.cc:482] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:36.298416 31981 raft_consensus.cc:3037] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:36.298416 31982 raft_consensus.cc:3037] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:36.301360 31982 raft_consensus.cc:504] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.301360 31981 raft_consensus.cc:504] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.301997 31982 leader_election.cc:302] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:36.301997 31981 leader_election.cc:302] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:36.302817 31981 leader_election.cc:288] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:36.303082 31982 leader_election.cc:288] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:36.303243 31985 raft_consensus.cc:2781] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:36.303270 31986 raft_consensus.cc:2781] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:36.310380 31986 raft_consensus.cc:686] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:36.310333 31985 raft_consensus.cc:686] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:36.311045 31986 consensus_queue.cc:237] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.311043 31985 consensus_queue.cc:237] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.311707 31982 ts_tablet_manager.cc:1398] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
I20240501 13:59:36.312368 31981 ts_tablet_manager.cc:1398] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.025s user 0.018s sys 0.004s
I20240501 13:59:36.312844 31981 tablet_bootstrap.cc:492] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:36.316188 31981 tablet_bootstrap.cc:654] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:36.316231 30973 catalog_manager.cc:5484] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:36.320120 31981 tablet_bootstrap.cc:492] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:36.320286 31981 ts_tablet_manager.cc:1367] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240501 13:59:36.320438 30973 catalog_manager.cc:5484] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:36.321213 31981 raft_consensus.cc:348] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.321398 31981 raft_consensus.cc:374] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:36.321491 31981 raft_consensus.cc:729] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:36.321746 31981 consensus_queue.cc:260] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.321947 31981 raft_consensus.cc:388] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:36.322029 31981 raft_consensus.cc:482] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:36.322149 31981 raft_consensus.cc:3037] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:36.324337 31981 raft_consensus.cc:504] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.324576 31981 leader_election.cc:302] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:36.324764 31981 leader_election.cc:288] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:36.324836 31986 raft_consensus.cc:2781] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:36.325038 31986 raft_consensus.cc:686] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:36.325217 31981 ts_tablet_manager.cc:1398] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 13:59:36.325266 31986 consensus_queue.cc:237] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.329351 30973 catalog_manager.cc:5484] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
W20240501 13:59:36.678364 31997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:36.678753 31997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:36.688369 31997 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 113.436 ms
time per row: 0.0567182 ms
I20240501 13:59:36.944643 30969 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43758:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:36.953508 31225 tablet_service.cc:1467] Processing CreateTablet for tablet 2af886c958f649af819ee997d78babed (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb2daa2611c4ac88c321041f5f617e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:36.953617 31224 tablet_service.cc:1467] Processing CreateTablet for tablet 71040610692d4cd3babce1e103b2bd08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb2daa2611c4ac88c321041f5f617e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:36.954034 31225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af886c958f649af819ee997d78babed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.953928 31093 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6e706bde6441d7b9fdbfe69731da5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=4fb2daa2611c4ac88c321041f5f617e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:36.954214 31224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71040610692d4cd3babce1e103b2bd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.954324 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6e706bde6441d7b9fdbfe69731da5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:36.959390 32040 tablet_bootstrap.cc:492] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: Bootstrap starting.
I20240501 13:59:36.960752 32041 tablet_bootstrap.cc:492] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:36.961733 32040 tablet_bootstrap.cc:654] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:36.962396 32041 tablet_bootstrap.cc:654] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:36.964407 32040 tablet_bootstrap.cc:492] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: No bootstrap required, opened a new log
I20240501 13:59:36.964525 32041 tablet_bootstrap.cc:492] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:36.964558 32040 ts_tablet_manager.cc:1367] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:36.964658 32041 ts_tablet_manager.cc:1367] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:36.965317 32041 raft_consensus.cc:348] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.965515 32041 raft_consensus.cc:374] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:36.965517 32040 raft_consensus.cc:348] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.965608 32041 raft_consensus.cc:729] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:36.965673 32040 raft_consensus.cc:374] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:36.965756 32040 raft_consensus.cc:729] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Initialized, Role: FOLLOWER
I20240501 13:59:36.965842 32041 consensus_queue.cc:260] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.966025 32041 raft_consensus.cc:388] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:36.965972 32040 consensus_queue.cc:260] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.966099 32041 raft_consensus.cc:482] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:36.966151 32040 raft_consensus.cc:388] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:36.966204 32041 raft_consensus.cc:3037] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:36.966226 32040 raft_consensus.cc:482] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:36.966334 32040 raft_consensus.cc:3037] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:36.968874 32041 raft_consensus.cc:504] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.968874 32040 raft_consensus.cc:504] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.969130 32040 leader_election.cc:302] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:36.969130 32041 leader_election.cc:302] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:36.969319 32040 leader_election.cc:288] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:36.969326 32041 leader_election.cc:288] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:36.969429 31986 raft_consensus.cc:2781] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:36.969806 32041 ts_tablet_manager.cc:1398] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 13:59:36.969882 32029 raft_consensus.cc:2781] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:36.970474 31986 raft_consensus.cc:686] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:36.970794 31986 consensus_queue.cc:237] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:36.970947 32040 ts_tablet_manager.cc:1398] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240501 13:59:36.971166 32040 tablet_bootstrap.cc:492] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7: Bootstrap starting.
I20240501 13:59:36.971519 32029 raft_consensus.cc:686] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:36.971858 32029 consensus_queue.cc:237] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.972690 32040 tablet_bootstrap.cc:654] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:36.974279 30969 catalog_manager.cc:5484] T 2d6e706bde6441d7b9fdbfe69731da5a P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:36.976637 32040 tablet_bootstrap.cc:492] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7: No bootstrap required, opened a new log
I20240501 13:59:36.976801 32040 ts_tablet_manager.cc:1367] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 13:59:36.977526 32040 raft_consensus.cc:348] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.977715 32040 raft_consensus.cc:374] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:36.977784 32040 raft_consensus.cc:729] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Initialized, Role: FOLLOWER
I20240501 13:59:36.977797 30970 catalog_manager.cc:5484] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:36.978006 32040 consensus_queue.cc:260] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.978169 32040 raft_consensus.cc:388] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:36.978267 32040 raft_consensus.cc:482] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:36.978358 32040 raft_consensus.cc:3037] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:36.980667 32040 raft_consensus.cc:504] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.980883 32040 leader_election.cc:302] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:36.981067 32040 leader_election.cc:288] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:36.981113 32029 raft_consensus.cc:2781] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:36.981302 32029 raft_consensus.cc:686] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:36.981526 32040 ts_tablet_manager.cc:1398] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 13:59:36.981529 32029 consensus_queue.cc:237] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:36.983610 30970 catalog_manager.cc:5484] T 71040610692d4cd3babce1e103b2bd08 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } health_report { overall_health: HEALTHY } } }
W20240501 13:59:37.319387 32051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:37.319705 32051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:37.327304 32051 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 116.966 ms
time per row: 0.0584828 ms
I20240501 13:59:37.587963 30970 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43758:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:37.596858 31093 tablet_service.cc:1467] Processing CreateTablet for tablet 226f3f612c1e4ca583c4b3a3e2a0c925 (DEFAULT_TABLE table=rebalance_test_table_2 [id=27639d7577424922b8eeeef214bcd4d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:37.597021 31224 tablet_service.cc:1467] Processing CreateTablet for tablet 97c6a2dd8f154b61810ae1aa0ee90b49 (DEFAULT_TABLE table=rebalance_test_table_2 [id=27639d7577424922b8eeeef214bcd4d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:37.597347 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226f3f612c1e4ca583c4b3a3e2a0c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:37.597394 31224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c6a2dd8f154b61810ae1aa0ee90b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:37.597977 31094 tablet_service.cc:1467] Processing CreateTablet for tablet 52793644a7654afaa9b07340d3c02c1b (DEFAULT_TABLE table=rebalance_test_table_2 [id=27639d7577424922b8eeeef214bcd4d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:37.598335 31094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52793644a7654afaa9b07340d3c02c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:37.603957 32093 tablet_bootstrap.cc:492] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:37.604281 32094 tablet_bootstrap.cc:492] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7: Bootstrap starting.
I20240501 13:59:37.605370 32093 tablet_bootstrap.cc:654] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:37.605533 32094 tablet_bootstrap.cc:654] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:37.607252 32094 tablet_bootstrap.cc:492] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7: No bootstrap required, opened a new log
I20240501 13:59:37.607182 32093 tablet_bootstrap.cc:492] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:37.607409 32094 ts_tablet_manager.cc:1367] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:37.607465 32093 ts_tablet_manager.cc:1367] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:37.608057 32094 raft_consensus.cc:348] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:37.608088 32093 raft_consensus.cc:348] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.608196 32094 raft_consensus.cc:374] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:37.608247 32093 raft_consensus.cc:374] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:37.608256 32094 raft_consensus.cc:729] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Initialized, Role: FOLLOWER
I20240501 13:59:37.608335 32093 raft_consensus.cc:729] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:37.608462 32094 consensus_queue.cc:260] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:37.608568 32093 consensus_queue.cc:260] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.608650 32094 raft_consensus.cc:388] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:37.608722 32094 raft_consensus.cc:482] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:37.608726 32093 raft_consensus.cc:388] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:37.608788 32093 raft_consensus.cc:482] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:37.608821 32094 raft_consensus.cc:3037] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:37.608887 32093 raft_consensus.cc:3037] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:37.610652 32094 raft_consensus.cc:504] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:37.610707 32093 raft_consensus.cc:504] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.610929 32093 leader_election.cc:302] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:37.610905 32094 leader_election.cc:302] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:37.611104 32094 leader_election.cc:288] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:37.611196 32044 raft_consensus.cc:2781] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:37.611249 31986 raft_consensus.cc:2781] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:37.611575 32094 ts_tablet_manager.cc:1398] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240501 13:59:37.611634 31986 raft_consensus.cc:686] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:37.611827 32044 raft_consensus.cc:686] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:37.611902 31986 consensus_queue.cc:237] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.612075 32044 consensus_queue.cc:237] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:37.612717 32093 leader_election.cc:288] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:37.613260 32093 ts_tablet_manager.cc:1398] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240501 13:59:37.613721 32093 tablet_bootstrap.cc:492] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:37.614629 30970 catalog_manager.cc:5484] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:37.615422 32093 tablet_bootstrap.cc:654] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:37.615558 30967 catalog_manager.cc:5484] T 97c6a2dd8f154b61810ae1aa0ee90b49 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:37.617811 32093 tablet_bootstrap.cc:492] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:37.618022 32093 ts_tablet_manager.cc:1367] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 13:59:37.618932 32093 raft_consensus.cc:348] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.619140 32093 raft_consensus.cc:374] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:37.619232 32093 raft_consensus.cc:729] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:37.619477 32093 consensus_queue.cc:260] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.619791 32093 raft_consensus.cc:388] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:37.619886 32093 raft_consensus.cc:482] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:37.619977 32093 raft_consensus.cc:3037] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:37.622354 32093 raft_consensus.cc:504] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.622644 32093 leader_election.cc:302] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:37.622913 32093 leader_election.cc:288] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:37.622949 31986 raft_consensus.cc:2781] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:37.623081 31986 raft_consensus.cc:686] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:37.623345 31986 consensus_queue.cc:237] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:37.625646 30970 catalog_manager.cc:5484] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:37.627127 32093 ts_tablet_manager.cc:1398] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
W20240501 13:59:37.963708 32103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:37.964030 32103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:37.972797 32103 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 113.841 ms
time per row: 0.0569205 ms
I20240501 13:59:38.227639 30965 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43758:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:38.236680 31224 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe5276e21884c3eb35aa88d0849aa59 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3ffcfa40e2b4ecdbdf18c2a3555b352]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:38.236805 31225 tablet_service.cc:1467] Processing CreateTablet for tablet ea43ba0cf5e14827b80fbd3962b6bdbb (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3ffcfa40e2b4ecdbdf18c2a3555b352]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:38.237118 31224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe5276e21884c3eb35aa88d0849aa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.237360 31225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea43ba0cf5e14827b80fbd3962b6bdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.237722 31094 tablet_service.cc:1467] Processing CreateTablet for tablet c77d9c76031c47aa994f1bf89139cb1d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3ffcfa40e2b4ecdbdf18c2a3555b352]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:38.238133 31094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77d9c76031c47aa994f1bf89139cb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.242928 32143 tablet_bootstrap.cc:492] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: Bootstrap starting.
I20240501 13:59:38.244267 32144 tablet_bootstrap.cc:492] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:38.244622 32143 tablet_bootstrap.cc:654] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:38.245745 32144 tablet_bootstrap.cc:654] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:38.246583 32143 tablet_bootstrap.cc:492] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: No bootstrap required, opened a new log
I20240501 13:59:38.246733 32143 ts_tablet_manager.cc:1367] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:38.247395 32143 raft_consensus.cc:348] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.247488 32144 tablet_bootstrap.cc:492] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:38.247543 32143 raft_consensus.cc:374] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:38.247611 32144 ts_tablet_manager.cc:1367] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:38.247609 32143 raft_consensus.cc:729] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Initialized, Role: FOLLOWER
I20240501 13:59:38.247812 32143 consensus_queue.cc:260] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.247951 32143 raft_consensus.cc:388] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:38.248020 32143 raft_consensus.cc:482] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:38.248121 32143 raft_consensus.cc:3037] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:38.248301 32144 raft_consensus.cc:348] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.248445 32144 raft_consensus.cc:374] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:38.248526 32144 raft_consensus.cc:729] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:38.248730 32144 consensus_queue.cc:260] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.248899 32144 raft_consensus.cc:388] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:38.248973 32144 raft_consensus.cc:482] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:38.249063 32144 raft_consensus.cc:3037] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:38.250185 32143 raft_consensus.cc:504] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.250393 32143 leader_election.cc:302] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:38.250572 32143 leader_election.cc:288] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:38.250651 32044 raft_consensus.cc:2781] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:38.251202 32144 raft_consensus.cc:504] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.251349 32143 ts_tablet_manager.cc:1398] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 13:59:38.251477 32144 leader_election.cc:302] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:38.251574 32044 raft_consensus.cc:686] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:38.251636 32143 tablet_bootstrap.cc:492] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: Bootstrap starting.
I20240501 13:59:38.251660 32144 leader_election.cc:288] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:38.251856 31986 raft_consensus.cc:2781] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:38.251833 32044 consensus_queue.cc:237] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.252252 31986 raft_consensus.cc:686] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:38.252521 31986 consensus_queue.cc:237] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.253430 32143 tablet_bootstrap.cc:654] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:38.255148 32144 ts_tablet_manager.cc:1398] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 13:59:38.256493 30965 catalog_manager.cc:5484] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:38.259871 32143 tablet_bootstrap.cc:492] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: No bootstrap required, opened a new log
I20240501 13:59:38.260074 32143 ts_tablet_manager.cc:1367] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240501 13:59:38.260933 32143 raft_consensus.cc:348] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.261150 32143 raft_consensus.cc:374] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:38.261231 32143 raft_consensus.cc:729] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Initialized, Role: FOLLOWER
I20240501 13:59:38.261497 32143 consensus_queue.cc:260] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.261662 32143 raft_consensus.cc:388] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:38.261765 32143 raft_consensus.cc:482] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:38.261860 32143 raft_consensus.cc:3037] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:38.264503 32143 raft_consensus.cc:504] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.264796 32143 leader_election.cc:302] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:38.265005 32143 leader_election.cc:288] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:38.264891 30968 catalog_manager.cc:5484] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:38.265453 32143 ts_tablet_manager.cc:1398] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 13:59:38.265677 32044 raft_consensus.cc:2781] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:38.266531 32044 raft_consensus.cc:686] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:38.266775 32044 consensus_queue.cc:237] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.269914 30968 catalog_manager.cc:5484] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } health_report { overall_health: HEALTHY } } }
W20240501 13:59:38.598635 32154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:38.598989 32154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:38.606622 32154 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 141.501 ms
time per row: 0.0707505 ms
I20240501 13:59:38.896051 30970 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43758:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 13:59:38.905306 31094 tablet_service.cc:1467] Processing CreateTablet for tablet 20337968dab04bea87be1f0557def220 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c758a9c63300499caf401bd3f77940bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:38.905308 31224 tablet_service.cc:1467] Processing CreateTablet for tablet f9834d3643f84f7dbedbca6bc836ed49 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c758a9c63300499caf401bd3f77940bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:38.905612 31093 tablet_service.cc:1467] Processing CreateTablet for tablet e642f36e29f842ac80dddcfda574623b (DEFAULT_TABLE table=rebalance_test_table_4 [id=c758a9c63300499caf401bd3f77940bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:38.905782 31094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20337968dab04bea87be1f0557def220. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.905787 31224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9834d3643f84f7dbedbca6bc836ed49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.906402 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e642f36e29f842ac80dddcfda574623b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:38.913847 32194 tablet_bootstrap.cc:492] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:38.915997 32194 tablet_bootstrap.cc:654] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:38.917816 32194 tablet_bootstrap.cc:492] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:38.917984 32194 ts_tablet_manager.cc:1367] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 13:59:38.918777 32195 tablet_bootstrap.cc:492] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: Bootstrap starting.
I20240501 13:59:38.918823 32194 raft_consensus.cc:348] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.919056 32194 raft_consensus.cc:374] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:38.919163 32194 raft_consensus.cc:729] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:38.919412 32194 consensus_queue.cc:260] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.919579 32194 raft_consensus.cc:388] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:38.919654 32194 raft_consensus.cc:482] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:38.919754 32194 raft_consensus.cc:3037] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:38.920811 32195 tablet_bootstrap.cc:654] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:38.921730 32194 raft_consensus.cc:504] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.922066 32194 leader_election.cc:302] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:38.922412 31986 raft_consensus.cc:2781] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:38.924274 32195 tablet_bootstrap.cc:492] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: No bootstrap required, opened a new log
I20240501 13:59:38.924461 32195 ts_tablet_manager.cc:1367] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 13:59:38.925410 32195 raft_consensus.cc:348] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.925623 32195 raft_consensus.cc:374] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:38.925719 32195 raft_consensus.cc:729] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Initialized, Role: FOLLOWER
I20240501 13:59:38.925998 32195 consensus_queue.cc:260] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.926142 32194 leader_election.cc:288] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:38.926311 32195 raft_consensus.cc:388] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:38.926419 32195 raft_consensus.cc:482] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:38.926596 32195 raft_consensus.cc:3037] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:38.929590 32195 raft_consensus.cc:504] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.929905 32195 leader_election.cc:302] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:38.930124 32195 leader_election.cc:288] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:38.930248 32044 raft_consensus.cc:2781] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:38.930660 32195 ts_tablet_manager.cc:1398] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240501 13:59:38.932634 32194 ts_tablet_manager.cc:1398] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240501 13:59:38.932941 32194 tablet_bootstrap.cc:492] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659: Bootstrap starting.
I20240501 13:59:38.934173 32194 tablet_bootstrap.cc:654] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:38.934804 32044 raft_consensus.cc:686] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:38.934995 31986 raft_consensus.cc:686] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:38.935053 32044 consensus_queue.cc:237] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } }
I20240501 13:59:38.935237 31986 consensus_queue.cc:237] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.936976 32194 tablet_bootstrap.cc:492] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659: No bootstrap required, opened a new log
I20240501 13:59:38.937131 32194 ts_tablet_manager.cc:1367] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:38.937749 32194 raft_consensus.cc:348] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.937932 32194 raft_consensus.cc:374] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:38.938009 32194 raft_consensus.cc:729] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Initialized, Role: FOLLOWER
I20240501 13:59:38.938040 30970 catalog_manager.cc:5484] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:38.938247 32194 consensus_queue.cc:260] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.938411 32194 raft_consensus.cc:388] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:38.938534 32194 raft_consensus.cc:482] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:38.938630 32194 raft_consensus.cc:3037] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:38.940613 32194 raft_consensus.cc:504] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.940882 32194 leader_election.cc:302] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:38.941151 32199 raft_consensus.cc:2781] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:38.941314 32199 raft_consensus.cc:686] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Leader. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:38.941570 32199 consensus_queue.cc:237] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } }
I20240501 13:59:38.943171 32194 leader_election.cc:288] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:38.943756 32194 ts_tablet_manager.cc:1398] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20240501 13:59:38.945088 30970 catalog_manager.cc:5484] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:38.949914 30970 catalog_manager.cc:5484] T e642f36e29f842ac80dddcfda574623b P f3c5452a71ac4778977f154db2cb1659 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c5452a71ac4778977f154db2cb1659 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } health_report { overall_health: HEALTHY } } }
W20240501 13:59:39.291751 32206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:39.292074 32206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:39.299571 32206 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 110.203 ms
time per row: 0.0551015 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 13:59:40.411573 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.3:37467
--local_ip_for_outbound_sockets=127.28.17.3
--tserver_master_addrs=127.28.17.62:46735
--webserver_port=35929
--webserver_interface=127.28.17.3
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:40.745720 32245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:40.746049 32245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:40.746177 32245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:40.754145 32245 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:40.754307 32245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:40.754462 32245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.3
I20240501 13:59:40.763937 32245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.3:37467
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.17.3
--webserver_port=35929
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.3
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:40.765303 32245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:40.767091 32245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:40.779870 32250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:40.780120 32253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:40.780771 32251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:40.781358 32245 server_base.cc:1034] running on GCE node
I20240501 13:59:40.781991 32245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:40.783032 32245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:40.784237 32245 hybrid_clock.cc:648] HybridClock initialized: now 1714571980784184 us; error 45 us; skew 500 ppm
I20240501 13:59:40.784575 32245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:40.786936 32245 webserver.cc:460] Webserver started at http://127.28.17.3:35929/ using document root <none> and password file <none>
I20240501 13:59:40.787688 32245 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:40.787811 32245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:40.793365 32245 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20240501 13:59:40.796389 32259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:40.797849 32245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:40.798015 32245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3fc17cbb4edf4c11b1d6132ae4896997"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-dx8c"
I20240501 13:59:40.798911 32245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:40.819691 32245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:40.820520 32245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:40.820830 32245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:40.821784 32245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:40.823324 32245 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:40.823401 32245 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:40.823521 32245 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:40.823578 32245 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:40.852537 32245 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.3:37467
I20240501 13:59:40.852615 32371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.3:37467 every 8 connection(s)
I20240501 13:59:40.854065 32245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:40.859632 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 32245
I20240501 13:59:40.860183 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.4:34759
--local_ip_for_outbound_sockets=127.28.17.4
--tserver_master_addrs=127.28.17.62:46735
--webserver_port=39511
--webserver_interface=127.28.17.4
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:40.867718 32372 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:40.868074 32372 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:40.869099 32372 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:40.870923 30969 ts_manager.cc:194] Re-registered known tserver with Master: 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3:37467)
I20240501 13:59:40.872078 30969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.3:42985
W20240501 13:59:41.190799 32376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:41.191078 32376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:41.191174 32376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:41.198714 32376 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:41.198839 32376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:41.198956 32376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.4
I20240501 13:59:41.208099 32376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.4:34759
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.17.4
--webserver_port=39511
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.4
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:41.209350 32376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:41.211047 32376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:41.222959 32381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:41.223460 32384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:41.224081 32382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:41.224772 32376 server_base.cc:1034] running on GCE node
I20240501 13:59:41.225445 32376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:41.226529 32376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:41.227766 32376 hybrid_clock.cc:648] HybridClock initialized: now 1714571981227712 us; error 51 us; skew 500 ppm
I20240501 13:59:41.228099 32376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:41.230414 32376 webserver.cc:460] Webserver started at http://127.28.17.4:39511/ using document root <none> and password file <none>
I20240501 13:59:41.231197 32376 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:41.231323 32376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:41.236649 32376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 13:59:41.239877 32390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:41.241389 32376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:41.241564 32376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d43e3c8c63554319b7bd3f1881889c32"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:41.242630 32376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:41.279981 32376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:41.280872 32376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:41.281157 32376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:41.282137 32376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:41.284106 32397 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 13:59:41.289433 32376 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 13:59:41.289556 32376 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 13:59:41.289711 32376 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 13:59:41.292580 32376 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 13:59:41.292690 32376 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:41.292999 32397 tablet_bootstrap.cc:492] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Bootstrap starting.
I20240501 13:59:41.312723 32397 log.cc:824] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:41.316651 32397 tablet_bootstrap.cc:492] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:41.317178 32397 tablet_bootstrap.cc:492] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Bootstrap complete.
I20240501 13:59:41.318123 32397 ts_tablet_manager.cc:1367] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.017s
I20240501 13:59:41.328356 32397 raft_consensus.cc:348] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:41.329174 32397 raft_consensus.cc:729] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43e3c8c63554319b7bd3f1881889c32, State: Initialized, Role: FOLLOWER
I20240501 13:59:41.330190 32397 consensus_queue.cc:260] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:41.330459 32397 raft_consensus.cc:388] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:41.330646 32397 raft_consensus.cc:482] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:41.330817 32397 raft_consensus.cc:3037] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:41.334471 32397 raft_consensus.cc:504] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:41.335206 32397 leader_election.cc:302] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43e3c8c63554319b7bd3f1881889c32; no voters:
I20240501 13:59:41.335922 32397 leader_election.cc:288] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 13:59:41.336063 32494 raft_consensus.cc:2781] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:41.337641 32494 raft_consensus.cc:686] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [term 2 LEADER]: Becoming Leader. State: Replica: d43e3c8c63554319b7bd3f1881889c32, State: Running, Role: LEADER
I20240501 13:59:41.338173 32397 ts_tablet_manager.cc:1398] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32: Time spent starting tablet: real 0.020s user 0.021s sys 0.000s
I20240501 13:59:41.338403 32494 consensus_queue.cc:237] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } }
I20240501 13:59:41.341051 32376 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.4:34759
I20240501 13:59:41.341092 32508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.4:34759 every 8 connection(s)
I20240501 13:59:41.342785 32376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:41.350096 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 32376
I20240501 13:59:41.350932 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:46587
--local_ip_for_outbound_sockets=127.28.17.5
--tserver_master_addrs=127.28.17.62:46735
--webserver_port=41955
--webserver_interface=127.28.17.5
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:41.355147 32510 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:41.355475 32510 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:41.356564 32510 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:41.359202 30969 ts_manager.cc:194] Re-registered known tserver with Master: d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4:34759)
I20240501 13:59:41.359586 30969 catalog_manager.cc:5484] T 81f2b5656f784e8caa4ade4c76228899 P d43e3c8c63554319b7bd3f1881889c32 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d43e3c8c63554319b7bd3f1881889c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:41.364588 30969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.4:58623
I20240501 13:59:41.367462 32510 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
W20240501 13:59:41.681717 32513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:41.682050 32513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:41.682169 32513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:41.689718 32513 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:41.689880 32513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:41.690032 32513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 13:59:41.699635 32513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:46587
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=41955
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:41.700965 32513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:41.702806 32513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:41.714910 32522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:41.715464 32523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:41.715981 32525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:41.716632 32513 server_base.cc:1034] running on GCE node
I20240501 13:59:41.717257 32513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:41.718262 32513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:41.719460 32513 hybrid_clock.cc:648] HybridClock initialized: now 1714571981719400 us; error 57 us; skew 500 ppm
I20240501 13:59:41.719785 32513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:41.726524 32513 webserver.cc:460] Webserver started at http://127.28.17.5:41955/ using document root <none> and password file <none>
I20240501 13:59:41.727408 32513 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:41.727548 32513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:41.733050 32513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 13:59:41.735936 32531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:41.737386 32513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:41.737568 32513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b6318df9141f4124938a82e74ce5fdcd"
format_stamp: "Formatted at 2024-05-01 13:59:30 on dist-test-slave-dx8c"
I20240501 13:59:41.738705 32513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:41.761224 32513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:41.762125 32513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:41.762423 32513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:41.763435 32513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:41.764948 32513 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:41.765020 32513 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:41.765130 32513 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:41.765177 32513 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:41.792742 32513 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:46587
I20240501 13:59:41.792760 32643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:46587 every 8 connection(s)
I20240501 13:59:41.794202 32513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:41.799700 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 32513
I20240501 13:59:41.800258 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.6:42705
--local_ip_for_outbound_sockets=127.28.17.6
--tserver_master_addrs=127.28.17.62:46735
--webserver_port=42707
--webserver_interface=127.28.17.6
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:41.807256 32644 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:41.807614 32644 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:41.808617 32644 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:41.810088 30969 ts_manager.cc:194] Re-registered known tserver with Master: b6318df9141f4124938a82e74ce5fdcd (127.28.17.5:46587)
I20240501 13:59:41.811013 30969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:48001
I20240501 13:59:41.875424 32372 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
W20240501 13:59:42.135886 32647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:42.136212 32647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:42.136351 32647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:42.143936 32647 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:42.144085 32647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:42.144263 32647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.6
I20240501 13:59:42.153618 32647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.6:42705
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.17.6
--webserver_port=42707
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.6
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:42.154991 32647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:42.156780 32647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:42.168735 32652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:42.169171 32655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:42.169816 32653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:42.170400 32647 server_base.cc:1034] running on GCE node
I20240501 13:59:42.171049 32647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:42.171993 32647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:42.173187 32647 hybrid_clock.cc:648] HybridClock initialized: now 1714571982173127 us; error 50 us; skew 500 ppm
I20240501 13:59:42.173507 32647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:42.175767 32647 webserver.cc:460] Webserver started at http://127.28.17.6:42707/ using document root <none> and password file <none>
I20240501 13:59:42.176518 32647 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:42.176645 32647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:42.181905 32647 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 13:59:42.184821 32661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:42.186185 32647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:42.186352 32647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9857990f442f4e7f98122344ebe75ffb"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:42.187216 32647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:42.201529 32647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:42.202389 32647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:42.202757 32647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:42.203790 32647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:42.205374 32647 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:42.205454 32647 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:42.205667 32647 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:42.205752 32647 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:42.234937 32647 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.6:42705
I20240501 13:59:42.234977 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.6:42705 every 8 connection(s)
I20240501 13:59:42.236497 32647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:42.238729 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 32647
I20240501 13:59:42.239432 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.7:45721
--local_ip_for_outbound_sockets=127.28.17.7
--tserver_master_addrs=127.28.17.62:46735
--webserver_port=33937
--webserver_interface=127.28.17.7
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:42.250696 306 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:42.251083 306 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:42.252194 306 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:42.254060 30969 ts_manager.cc:194] Re-registered known tserver with Master: 9857990f442f4e7f98122344ebe75ffb (127.28.17.6:42705)
I20240501 13:59:42.254961 30969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.6:57219
W20240501 13:59:42.579141 309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:42.579442 309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:42.579573 309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:42.587092 309 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:42.587234 309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:42.587379 309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.7
I20240501 13:59:42.597139 309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.7:45721
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.17.7
--webserver_port=33937
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.7
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:42.598493 309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:42.600337 309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:42.613279 315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:42.613602 314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:42.614060 317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:42.614553 309 server_base.cc:1034] running on GCE node
I20240501 13:59:42.615165 309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:42.616168 309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:42.617410 309 hybrid_clock.cc:648] HybridClock initialized: now 1714571982617357 us; error 49 us; skew 500 ppm
I20240501 13:59:42.618232 309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:42.620656 309 webserver.cc:460] Webserver started at http://127.28.17.7:33937/ using document root <none> and password file <none>
I20240501 13:59:42.621510 309 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:42.621639 309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:42.627285 309 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 13:59:42.630260 323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:42.631772 309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 13:59:42.631949 309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "92a0ab1d123c4dcba63e5e5064e059f6"
format_stamp: "Formatted at 2024-05-01 13:59:31 on dist-test-slave-dx8c"
I20240501 13:59:42.632849 309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:42.658368 309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:42.659247 309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:42.659538 309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:42.660521 309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:42.662132 309 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:42.662220 309 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:42.662341 309 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:42.662392 309 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:42.697307 309 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.7:45721
I20240501 13:59:42.697352 435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.7:45721 every 8 connection(s)
I20240501 13:59:42.698827 309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:42.701107 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 309
I20240501 13:59:42.714263 436 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:42.714624 436 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:42.715456 436 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:42.717039 30969 ts_manager.cc:194] Re-registered known tserver with Master: 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7:45721)
I20240501 13:59:42.718155 30969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.7:49245
I20240501 13:59:42.814143 32644 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
I20240501 13:59:43.232831 371 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.238137 32579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.238339 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:43.247592 31224 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:43.252307 32307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.258392 306 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
I20240501 13:59:43.269233 31094 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 13:59:43.292666 32709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.468281 32307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.470438 31224 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 13:59:43.473698 371 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.493103 32709 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.508177 32579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:43.532444 32439 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 13:59:43.533068 31094 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 13:59:43.605715 31245 consensus_queue.cc:237] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } }
I20240501 13:59:43.608390 526 raft_consensus.cc:2932] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } } }
I20240501 13:59:43.613960 30966 catalog_manager.cc:5484] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:43.617198 31179 consensus_peers.cc:485] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 -> Peer 9857990f442f4e7f98122344ebe75ffb (127.28.17.6:42705): Couldn't send request to peer 9857990f442f4e7f98122344ebe75ffb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fe5276e21884c3eb35aa88d0849aa59. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.626206 31114 consensus_queue.cc:237] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } }
I20240501 13:59:43.628773 539 raft_consensus.cc:2932] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } } }
I20240501 13:59:43.635762 30968 catalog_manager.cc:5484] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index -1 to 4, NON_VOTER 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:43.638293 31050 consensus_peers.cc:485] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 -> Peer 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3:37467): Couldn't send request to peer 3fc17cbb4edf4c11b1d6132ae4896997. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4740a81166714d6bb90c11a657f0a8d9. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.647307 31245 consensus_queue.cc:237] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } }
I20240501 13:59:43.648962 526 raft_consensus.cc:2932] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } } }
I20240501 13:59:43.653584 30968 catalog_manager.cc:5484] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index -1 to 4, NON_VOTER b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:43.655964 31181 consensus_peers.cc:485] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 -> Peer b6318df9141f4124938a82e74ce5fdcd (127.28.17.5:46587): Couldn't send request to peer b6318df9141f4124938a82e74ce5fdcd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea43ba0cf5e14827b80fbd3962b6bdbb. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.665355 31114 consensus_queue.cc:237] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } }
I20240501 13:59:43.667524 539 raft_consensus.cc:2932] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } } }
I20240501 13:59:43.672520 30968 catalog_manager.cc:5484] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index -1 to 4, NON_VOTER 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:43.674767 31050 consensus_peers.cc:485] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 -> Peer 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7:45721): Couldn't send request to peer 92a0ab1d123c4dcba63e5e5064e059f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 226f3f612c1e4ca583c4b3a3e2a0c925. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.687228 31245 consensus_queue.cc:237] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: true } }
I20240501 13:59:43.688970 526 raft_consensus.cc:2932] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: true } } }
I20240501 13:59:43.693902 30968 catalog_manager.cc:5484] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index -1 to 4, NON_VOTER d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:43.696120 31180 consensus_peers.cc:485] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 -> Peer d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4:34759): Couldn't send request to peer d43e3c8c63554319b7bd3f1881889c32. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2af886c958f649af819ee997d78babed. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.707056 31245 consensus_queue.cc:237] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } }
I20240501 13:59:43.708670 526 raft_consensus.cc:2932] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } } }
W20240501 13:59:43.709306 31179 consensus_peers.cc:485] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 -> Peer 9857990f442f4e7f98122344ebe75ffb (127.28.17.6:42705): Couldn't send request to peer 9857990f442f4e7f98122344ebe75ffb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9834d3643f84f7dbedbca6bc836ed49. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.712905 30968 catalog_manager.cc:5484] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:43.720784 436 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
I20240501 13:59:43.723520 31114 consensus_queue.cc:237] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } }
I20240501 13:59:43.725077 539 raft_consensus.cc:2932] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } } }
W20240501 13:59:43.725610 31050 consensus_peers.cc:485] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 -> Peer 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3:37467): Couldn't send request to peer 3fc17cbb4edf4c11b1d6132ae4896997. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52793644a7654afaa9b07340d3c02c1b. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.729916 30968 catalog_manager.cc:5484] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index -1 to 4, NON_VOTER 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:43.740353 31114 consensus_queue.cc:237] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } }
I20240501 13:59:43.742126 539 raft_consensus.cc:2932] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } } }
I20240501 13:59:43.747578 30968 catalog_manager.cc:5484] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index -1 to 4, NON_VOTER b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:43.748232 31050 consensus_peers.cc:485] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 -> Peer b6318df9141f4124938a82e74ce5fdcd (127.28.17.5:46587): Couldn't send request to peer b6318df9141f4124938a82e74ce5fdcd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20337968dab04bea87be1f0557def220. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.759101 31114 consensus_queue.cc:237] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } }
I20240501 13:59:43.760663 539 raft_consensus.cc:2932] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } } }
W20240501 13:59:43.761139 31050 consensus_peers.cc:485] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 -> Peer 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7:45721): Couldn't send request to peer 92a0ab1d123c4dcba63e5e5064e059f6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b71845f2e464353806c6306e4799456. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:43.765069 30968 catalog_manager.cc:5484] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index -1 to 4, NON_VOTER 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:44.081965 643 ts_tablet_manager.cc:902] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: Initiating tablet copy from peer 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:44.083765 643 tablet_copy_client.cc:308] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:36045
I20240501 13:59:44.092264 31265 tablet_copy_service.cc:140] P 73ddc7b55a734843abb88e9f6d2be4b7: Received BeginTabletCopySession request for tablet 2af886c958f649af819ee997d78babed from peer d43e3c8c63554319b7bd3f1881889c32 ({username='slave'} at 127.28.17.4:55573)
I20240501 13:59:44.092533 31265 tablet_copy_service.cc:161] P 73ddc7b55a734843abb88e9f6d2be4b7: Beginning new tablet copy session on tablet 2af886c958f649af819ee997d78babed from peer d43e3c8c63554319b7bd3f1881889c32 at {username='slave'} at 127.28.17.4:55573: session id = d43e3c8c63554319b7bd3f1881889c32-2af886c958f649af819ee997d78babed
I20240501 13:59:44.094038 655 ts_tablet_manager.cc:902] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: Initiating tablet copy from peer f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:44.095595 31265 tablet_copy_source_session.cc:205] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.097584 655 tablet_copy_client.cc:308] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:44235
I20240501 13:59:44.098762 643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af886c958f649af819ee997d78babed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.106918 643 tablet_copy_client.cc:783] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.107291 643 tablet_copy_client.cc:647] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.108141 31134 tablet_copy_service.cc:140] P f3c5452a71ac4778977f154db2cb1659: Received BeginTabletCopySession request for tablet 4740a81166714d6bb90c11a657f0a8d9 from peer 3fc17cbb4edf4c11b1d6132ae4896997 ({username='slave'} at 127.28.17.3:46805)
I20240501 13:59:44.108392 31134 tablet_copy_service.cc:161] P f3c5452a71ac4778977f154db2cb1659: Beginning new tablet copy session on tablet 4740a81166714d6bb90c11a657f0a8d9 from peer 3fc17cbb4edf4c11b1d6132ae4896997 at {username='slave'} at 127.28.17.3:46805: session id = 3fc17cbb4edf4c11b1d6132ae4896997-4740a81166714d6bb90c11a657f0a8d9
I20240501 13:59:44.109891 643 tablet_copy_client.cc:523] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.111450 31134 tablet_copy_source_session.cc:205] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.111485 663 ts_tablet_manager.cc:902] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: Initiating tablet copy from peer f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:44.114845 643 tablet_bootstrap.cc:492] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: Bootstrap starting.
I20240501 13:59:44.117671 655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4740a81166714d6bb90c11a657f0a8d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.135553 655 tablet_copy_client.cc:783] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.136087 655 tablet_copy_client.cc:647] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.139254 655 tablet_copy_client.cc:523] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.140671 663 tablet_copy_client.cc:308] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:44235
I20240501 13:59:44.142319 31134 tablet_copy_service.cc:140] P f3c5452a71ac4778977f154db2cb1659: Received BeginTabletCopySession request for tablet 52793644a7654afaa9b07340d3c02c1b from peer 3fc17cbb4edf4c11b1d6132ae4896997 ({username='slave'} at 127.28.17.3:46805)
I20240501 13:59:44.142576 31134 tablet_copy_service.cc:161] P f3c5452a71ac4778977f154db2cb1659: Beginning new tablet copy session on tablet 52793644a7654afaa9b07340d3c02c1b from peer 3fc17cbb4edf4c11b1d6132ae4896997 at {username='slave'} at 127.28.17.3:46805: session id = 3fc17cbb4edf4c11b1d6132ae4896997-52793644a7654afaa9b07340d3c02c1b
I20240501 13:59:44.145551 31134 tablet_copy_source_session.cc:205] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.147097 663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52793644a7654afaa9b07340d3c02c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.155009 663 tablet_copy_client.cc:783] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.155392 663 tablet_copy_client.cc:647] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.156087 655 tablet_bootstrap.cc:492] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: Bootstrap starting.
I20240501 13:59:44.158051 663 tablet_copy_client.cc:523] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.158816 673 ts_tablet_manager.cc:902] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: Initiating tablet copy from peer 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:44.162015 663 tablet_bootstrap.cc:492] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: Bootstrap starting.
I20240501 13:59:44.163015 674 ts_tablet_manager.cc:902] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: Initiating tablet copy from peer 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:44.176208 663 log.cc:824] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:44.187106 643 tablet_bootstrap.cc:492] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.189587 643 tablet_bootstrap.cc:492] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: Bootstrap complete.
I20240501 13:59:44.190551 643 ts_tablet_manager.cc:1367] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: Time spent bootstrapping tablet: real 0.076s user 0.053s sys 0.013s
I20240501 13:59:44.191067 681 ts_tablet_manager.cc:902] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: Initiating tablet copy from peer 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:44.191947 643 raft_consensus.cc:348] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: true } }
I20240501 13:59:44.192327 643 raft_consensus.cc:729] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d43e3c8c63554319b7bd3f1881889c32, State: Initialized, Role: LEARNER
I20240501 13:59:44.192615 643 consensus_queue.cc:260] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: true } }
I20240501 13:59:44.197072 674 tablet_copy_client.cc:308] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:36045
I20240501 13:59:44.204913 673 tablet_copy_client.cc:308] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:36045
I20240501 13:59:44.234277 697 ts_tablet_manager.cc:902] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: Initiating tablet copy from peer f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:44.235985 699 ts_tablet_manager.cc:902] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: Initiating tablet copy from peer f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:44.251967 31265 tablet_copy_service.cc:140] P 73ddc7b55a734843abb88e9f6d2be4b7: Received BeginTabletCopySession request for tablet f9834d3643f84f7dbedbca6bc836ed49 from peer 9857990f442f4e7f98122344ebe75ffb ({username='slave'} at 127.28.17.6:35655)
I20240501 13:59:44.252200 31265 tablet_copy_service.cc:161] P 73ddc7b55a734843abb88e9f6d2be4b7: Beginning new tablet copy session on tablet f9834d3643f84f7dbedbca6bc836ed49 from peer 9857990f442f4e7f98122344ebe75ffb at {username='slave'} at 127.28.17.6:35655: session id = 9857990f442f4e7f98122344ebe75ffb-f9834d3643f84f7dbedbca6bc836ed49
I20240501 13:59:44.255447 31265 tablet_copy_source_session.cc:205] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.261493 673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9834d3643f84f7dbedbca6bc836ed49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.269380 31265 tablet_copy_service.cc:140] P 73ddc7b55a734843abb88e9f6d2be4b7: Received BeginTabletCopySession request for tablet ea43ba0cf5e14827b80fbd3962b6bdbb from peer b6318df9141f4124938a82e74ce5fdcd ({username='slave'} at 127.28.17.5:60711)
I20240501 13:59:44.269678 31265 tablet_copy_service.cc:161] P 73ddc7b55a734843abb88e9f6d2be4b7: Beginning new tablet copy session on tablet ea43ba0cf5e14827b80fbd3962b6bdbb from peer b6318df9141f4124938a82e74ce5fdcd at {username='slave'} at 127.28.17.5:60711: session id = b6318df9141f4124938a82e74ce5fdcd-ea43ba0cf5e14827b80fbd3962b6bdbb
I20240501 13:59:44.272938 31265 tablet_copy_source_session.cc:205] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.277979 681 tablet_copy_client.cc:308] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:36045
I20240501 13:59:44.278928 31265 tablet_copy_service.cc:140] P 73ddc7b55a734843abb88e9f6d2be4b7: Received BeginTabletCopySession request for tablet 6fe5276e21884c3eb35aa88d0849aa59 from peer 9857990f442f4e7f98122344ebe75ffb ({username='slave'} at 127.28.17.6:35655)
I20240501 13:59:44.279202 31265 tablet_copy_service.cc:161] P 73ddc7b55a734843abb88e9f6d2be4b7: Beginning new tablet copy session on tablet 6fe5276e21884c3eb35aa88d0849aa59 from peer 9857990f442f4e7f98122344ebe75ffb at {username='slave'} at 127.28.17.6:35655: session id = 9857990f442f4e7f98122344ebe75ffb-6fe5276e21884c3eb35aa88d0849aa59
I20240501 13:59:44.281814 663 tablet_bootstrap.cc:492] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.282546 663 tablet_bootstrap.cc:492] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: Bootstrap complete.
I20240501 13:59:44.283123 663 ts_tablet_manager.cc:1367] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: Time spent bootstrapping tablet: real 0.121s user 0.064s sys 0.006s
I20240501 13:59:44.283458 699 tablet_copy_client.cc:308] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:44235
I20240501 13:59:44.283473 31265 tablet_copy_source_session.cc:205] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.284104 674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea43ba0cf5e14827b80fbd3962b6bdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.290582 663 raft_consensus.cc:348] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } }
I20240501 13:59:44.291345 663 raft_consensus.cc:729] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fc17cbb4edf4c11b1d6132ae4896997, State: Initialized, Role: LEARNER
I20240501 13:59:44.292083 663 consensus_queue.cc:260] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } }
I20240501 13:59:44.292963 697 tablet_copy_client.cc:308] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:44235
I20240501 13:59:44.295538 643 ts_tablet_manager.cc:1398] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32: Time spent starting tablet: real 0.105s user 0.020s sys 0.045s
I20240501 13:59:44.296595 31264 tablet_copy_service.cc:342] P 73ddc7b55a734843abb88e9f6d2be4b7: Request end of tablet copy session d43e3c8c63554319b7bd3f1881889c32-2af886c958f649af819ee997d78babed received from {username='slave'} at 127.28.17.4:55573
I20240501 13:59:44.296758 31264 tablet_copy_service.cc:433] P 73ddc7b55a734843abb88e9f6d2be4b7: ending tablet copy session d43e3c8c63554319b7bd3f1881889c32-2af886c958f649af819ee997d78babed on tablet 2af886c958f649af819ee997d78babed with peer d43e3c8c63554319b7bd3f1881889c32
I20240501 13:59:44.301033 655 tablet_bootstrap.cc:492] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.301617 655 tablet_bootstrap.cc:492] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: Bootstrap complete.
I20240501 13:59:44.301826 681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe5276e21884c3eb35aa88d0849aa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.302054 655 ts_tablet_manager.cc:1367] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: Time spent bootstrapping tablet: real 0.146s user 0.047s sys 0.008s
I20240501 13:59:44.303094 655 raft_consensus.cc:348] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } }
I20240501 13:59:44.303421 655 raft_consensus.cc:729] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fc17cbb4edf4c11b1d6132ae4896997, State: Initialized, Role: LEARNER
I20240501 13:59:44.303675 655 consensus_queue.cc:260] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: true } }
I20240501 13:59:44.307310 663 ts_tablet_manager.cc:1398] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997: Time spent starting tablet: real 0.024s user 0.010s sys 0.008s
I20240501 13:59:44.308765 655 ts_tablet_manager.cc:1398] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240501 13:59:44.309744 31134 tablet_copy_service.cc:342] P f3c5452a71ac4778977f154db2cb1659: Request end of tablet copy session 3fc17cbb4edf4c11b1d6132ae4896997-52793644a7654afaa9b07340d3c02c1b received from {username='slave'} at 127.28.17.3:46805
I20240501 13:59:44.309880 31134 tablet_copy_service.cc:433] P f3c5452a71ac4778977f154db2cb1659: ending tablet copy session 3fc17cbb4edf4c11b1d6132ae4896997-52793644a7654afaa9b07340d3c02c1b on tablet 52793644a7654afaa9b07340d3c02c1b with peer 3fc17cbb4edf4c11b1d6132ae4896997
I20240501 13:59:44.310530 31133 tablet_copy_service.cc:342] P f3c5452a71ac4778977f154db2cb1659: Request end of tablet copy session 3fc17cbb4edf4c11b1d6132ae4896997-4740a81166714d6bb90c11a657f0a8d9 received from {username='slave'} at 127.28.17.3:46805
I20240501 13:59:44.310634 31133 tablet_copy_service.cc:433] P f3c5452a71ac4778977f154db2cb1659: ending tablet copy session 3fc17cbb4edf4c11b1d6132ae4896997-4740a81166714d6bb90c11a657f0a8d9 on tablet 4740a81166714d6bb90c11a657f0a8d9 with peer 3fc17cbb4edf4c11b1d6132ae4896997
I20240501 13:59:44.319619 31133 tablet_copy_service.cc:140] P f3c5452a71ac4778977f154db2cb1659: Received BeginTabletCopySession request for tablet 226f3f612c1e4ca583c4b3a3e2a0c925 from peer 92a0ab1d123c4dcba63e5e5064e059f6 ({username='slave'} at 127.28.17.7:52413)
I20240501 13:59:44.319806 31133 tablet_copy_service.cc:161] P f3c5452a71ac4778977f154db2cb1659: Beginning new tablet copy session on tablet 226f3f612c1e4ca583c4b3a3e2a0c925 from peer 92a0ab1d123c4dcba63e5e5064e059f6 at {username='slave'} at 127.28.17.7:52413: session id = 92a0ab1d123c4dcba63e5e5064e059f6-226f3f612c1e4ca583c4b3a3e2a0c925
I20240501 13:59:44.322619 31133 tablet_copy_source_session.cc:205] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.319674 31134 tablet_copy_service.cc:140] P f3c5452a71ac4778977f154db2cb1659: Received BeginTabletCopySession request for tablet 6b71845f2e464353806c6306e4799456 from peer 92a0ab1d123c4dcba63e5e5064e059f6 ({username='slave'} at 127.28.17.7:52413)
I20240501 13:59:44.323225 31134 tablet_copy_service.cc:161] P f3c5452a71ac4778977f154db2cb1659: Beginning new tablet copy session on tablet 6b71845f2e464353806c6306e4799456 from peer 92a0ab1d123c4dcba63e5e5064e059f6 at {username='slave'} at 127.28.17.7:52413: session id = 92a0ab1d123c4dcba63e5e5064e059f6-6b71845f2e464353806c6306e4799456
I20240501 13:59:44.325992 31134 tablet_copy_source_session.cc:205] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.327435 699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226f3f612c1e4ca583c4b3a3e2a0c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.327867 697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b71845f2e464353806c6306e4799456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.329764 674 tablet_copy_client.cc:783] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.330147 674 tablet_copy_client.cc:647] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.331832 681 tablet_copy_client.cc:783] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.332882 681 tablet_copy_client.cc:647] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.336359 673 tablet_copy_client.cc:783] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.336797 674 tablet_copy_client.cc:523] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.339217 673 tablet_copy_client.cc:647] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.338980 681 tablet_copy_client.cc:523] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.341605 674 tablet_bootstrap.cc:492] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: Bootstrap starting.
I20240501 13:59:44.341955 699 tablet_copy_client.cc:783] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.342324 699 tablet_copy_client.cc:647] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.344659 699 tablet_copy_client.cc:523] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.344785 697 tablet_copy_client.cc:783] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.345178 697 tablet_copy_client.cc:647] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.347333 697 tablet_copy_client.cc:523] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.349306 673 tablet_copy_client.cc:523] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.354669 697 tablet_bootstrap.cc:492] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: Bootstrap starting.
I20240501 13:59:44.356935 726 ts_tablet_manager.cc:902] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: Initiating tablet copy from peer f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:44.359606 674 log.cc:824] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:44.360926 681 tablet_bootstrap.cc:492] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: Bootstrap starting.
I20240501 13:59:44.362882 699 tablet_bootstrap.cc:492] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: Bootstrap starting.
I20240501 13:59:44.363314 673 tablet_bootstrap.cc:492] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: Bootstrap starting.
I20240501 13:59:44.372152 699 log.cc:824] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:44.393882 673 log.cc:824] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:44.422200 674 tablet_bootstrap.cc:492] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.422904 674 tablet_bootstrap.cc:492] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: Bootstrap complete.
I20240501 13:59:44.423462 674 ts_tablet_manager.cc:1367] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: Time spent bootstrapping tablet: real 0.082s user 0.062s sys 0.006s
I20240501 13:59:44.425415 726 tablet_copy_client.cc:308] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:44235
I20240501 13:59:44.434578 31134 tablet_copy_service.cc:140] P f3c5452a71ac4778977f154db2cb1659: Received BeginTabletCopySession request for tablet 20337968dab04bea87be1f0557def220 from peer b6318df9141f4124938a82e74ce5fdcd ({username='slave'} at 127.28.17.5:42965)
I20240501 13:59:44.434793 31134 tablet_copy_service.cc:161] P f3c5452a71ac4778977f154db2cb1659: Beginning new tablet copy session on tablet 20337968dab04bea87be1f0557def220 from peer b6318df9141f4124938a82e74ce5fdcd at {username='slave'} at 127.28.17.5:42965: session id = b6318df9141f4124938a82e74ce5fdcd-20337968dab04bea87be1f0557def220
I20240501 13:59:44.437739 31134 tablet_copy_source_session.cc:205] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:44.439632 726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20337968dab04bea87be1f0557def220. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:44.440886 699 tablet_bootstrap.cc:492] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.440889 697 tablet_bootstrap.cc:492] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.441629 697 tablet_bootstrap.cc:492] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: Bootstrap complete.
I20240501 13:59:44.442247 697 ts_tablet_manager.cc:1367] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: Time spent bootstrapping tablet: real 0.088s user 0.050s sys 0.013s
I20240501 13:59:44.441638 699 tablet_bootstrap.cc:492] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: Bootstrap complete.
I20240501 13:59:44.442763 699 ts_tablet_manager.cc:1367] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: Time spent bootstrapping tablet: real 0.080s user 0.050s sys 0.013s
I20240501 13:59:44.447387 726 tablet_copy_client.cc:783] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:44.447152 674 raft_consensus.cc:348] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } }
I20240501 13:59:44.447710 726 tablet_copy_client.cc:647] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:44.447763 674 raft_consensus.cc:729] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6318df9141f4124938a82e74ce5fdcd, State: Initialized, Role: LEARNER
I20240501 13:59:44.448470 674 consensus_queue.cc:260] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } }
I20240501 13:59:44.450009 726 tablet_copy_client.cc:523] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:44.450601 697 raft_consensus.cc:348] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } }
I20240501 13:59:44.451105 697 raft_consensus.cc:729] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92a0ab1d123c4dcba63e5e5064e059f6, State: Initialized, Role: LEARNER
I20240501 13:59:44.451715 697 consensus_queue.cc:260] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } }
I20240501 13:59:44.452392 699 raft_consensus.cc:348] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } }
I20240501 13:59:44.452584 699 raft_consensus.cc:729] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92a0ab1d123c4dcba63e5e5064e059f6, State: Initialized, Role: LEARNER
I20240501 13:59:44.452806 699 consensus_queue.cc:260] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: true } }
I20240501 13:59:44.453840 726 tablet_bootstrap.cc:492] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: Bootstrap starting.
I20240501 13:59:44.468250 697 ts_tablet_manager.cc:1398] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6: Time spent starting tablet: real 0.026s user 0.007s sys 0.003s
I20240501 13:59:44.469344 31134 tablet_copy_service.cc:342] P f3c5452a71ac4778977f154db2cb1659: Request end of tablet copy session 92a0ab1d123c4dcba63e5e5064e059f6-6b71845f2e464353806c6306e4799456 received from {username='slave'} at 127.28.17.7:52413
I20240501 13:59:44.469497 31134 tablet_copy_service.cc:433] P f3c5452a71ac4778977f154db2cb1659: ending tablet copy session 92a0ab1d123c4dcba63e5e5064e059f6-6b71845f2e464353806c6306e4799456 on tablet 6b71845f2e464353806c6306e4799456 with peer 92a0ab1d123c4dcba63e5e5064e059f6
I20240501 13:59:44.471642 699 ts_tablet_manager.cc:1398] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6: Time spent starting tablet: real 0.029s user 0.007s sys 0.007s
I20240501 13:59:44.472318 31134 tablet_copy_service.cc:342] P f3c5452a71ac4778977f154db2cb1659: Request end of tablet copy session 92a0ab1d123c4dcba63e5e5064e059f6-226f3f612c1e4ca583c4b3a3e2a0c925 received from {username='slave'} at 127.28.17.7:52413
I20240501 13:59:44.472471 31134 tablet_copy_service.cc:433] P f3c5452a71ac4778977f154db2cb1659: ending tablet copy session 92a0ab1d123c4dcba63e5e5064e059f6-226f3f612c1e4ca583c4b3a3e2a0c925 on tablet 226f3f612c1e4ca583c4b3a3e2a0c925 with peer 92a0ab1d123c4dcba63e5e5064e059f6
I20240501 13:59:44.477067 674 ts_tablet_manager.cc:1398] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd: Time spent starting tablet: real 0.053s user 0.019s sys 0.009s
I20240501 13:59:44.477965 31265 tablet_copy_service.cc:342] P 73ddc7b55a734843abb88e9f6d2be4b7: Request end of tablet copy session b6318df9141f4124938a82e74ce5fdcd-ea43ba0cf5e14827b80fbd3962b6bdbb received from {username='slave'} at 127.28.17.5:60711
I20240501 13:59:44.478103 31265 tablet_copy_service.cc:433] P 73ddc7b55a734843abb88e9f6d2be4b7: ending tablet copy session b6318df9141f4124938a82e74ce5fdcd-ea43ba0cf5e14827b80fbd3962b6bdbb on tablet ea43ba0cf5e14827b80fbd3962b6bdbb with peer b6318df9141f4124938a82e74ce5fdcd
I20240501 13:59:44.478878 681 tablet_bootstrap.cc:492] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.479487 681 tablet_bootstrap.cc:492] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: Bootstrap complete.
I20240501 13:59:44.480821 681 ts_tablet_manager.cc:1367] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: Time spent bootstrapping tablet: real 0.120s user 0.055s sys 0.005s
I20240501 13:59:44.481645 32327 raft_consensus.cc:1203] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.484329 673 tablet_bootstrap.cc:492] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.484791 673 tablet_bootstrap.cc:492] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: Bootstrap complete.
I20240501 13:59:44.492285 673 ts_tablet_manager.cc:1367] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: Time spent bootstrapping tablet: real 0.129s user 0.043s sys 0.011s
I20240501 13:59:44.503768 673 raft_consensus.cc:348] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } }
I20240501 13:59:44.504312 673 raft_consensus.cc:729] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9857990f442f4e7f98122344ebe75ffb, State: Initialized, Role: LEARNER
I20240501 13:59:44.504911 673 consensus_queue.cc:260] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } }
I20240501 13:59:44.507191 673 ts_tablet_manager.cc:1398] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb: Time spent starting tablet: real 0.015s user 0.003s sys 0.001s
I20240501 13:59:44.508111 31265 tablet_copy_service.cc:342] P 73ddc7b55a734843abb88e9f6d2be4b7: Request end of tablet copy session 9857990f442f4e7f98122344ebe75ffb-f9834d3643f84f7dbedbca6bc836ed49 received from {username='slave'} at 127.28.17.6:35655
I20240501 13:59:44.508244 31265 tablet_copy_service.cc:433] P 73ddc7b55a734843abb88e9f6d2be4b7: ending tablet copy session 9857990f442f4e7f98122344ebe75ffb-f9834d3643f84f7dbedbca6bc836ed49 on tablet f9834d3643f84f7dbedbca6bc836ed49 with peer 9857990f442f4e7f98122344ebe75ffb
I20240501 13:59:44.503762 681 raft_consensus.cc:348] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } }
I20240501 13:59:44.509282 681 raft_consensus.cc:729] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9857990f442f4e7f98122344ebe75ffb, State: Initialized, Role: LEARNER
I20240501 13:59:44.509537 681 consensus_queue.cc:260] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: true } }
I20240501 13:59:44.514998 681 ts_tablet_manager.cc:1398] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb: Time spent starting tablet: real 0.034s user 0.005s sys 0.008s
I20240501 13:59:44.516042 31265 tablet_copy_service.cc:342] P 73ddc7b55a734843abb88e9f6d2be4b7: Request end of tablet copy session 9857990f442f4e7f98122344ebe75ffb-6fe5276e21884c3eb35aa88d0849aa59 received from {username='slave'} at 127.28.17.6:35655
I20240501 13:59:44.516185 31265 tablet_copy_service.cc:433] P 73ddc7b55a734843abb88e9f6d2be4b7: ending tablet copy session 9857990f442f4e7f98122344ebe75ffb-6fe5276e21884c3eb35aa88d0849aa59 on tablet 6fe5276e21884c3eb35aa88d0849aa59 with peer 9857990f442f4e7f98122344ebe75ffb
I20240501 13:59:44.524842 726 tablet_bootstrap.cc:492] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:44.525274 726 tablet_bootstrap.cc:492] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: Bootstrap complete.
I20240501 13:59:44.525588 726 ts_tablet_manager.cc:1367] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: Time spent bootstrapping tablet: real 0.072s user 0.040s sys 0.006s
I20240501 13:59:44.526602 726 raft_consensus.cc:348] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } }
I20240501 13:59:44.526829 726 raft_consensus.cc:729] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6318df9141f4124938a82e74ce5fdcd, State: Initialized, Role: LEARNER
I20240501 13:59:44.527009 726 consensus_queue.cc:260] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: true } }
I20240501 13:59:44.527524 726 ts_tablet_manager.cc:1398] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 13:59:44.528458 31134 tablet_copy_service.cc:342] P f3c5452a71ac4778977f154db2cb1659: Request end of tablet copy session b6318df9141f4124938a82e74ce5fdcd-20337968dab04bea87be1f0557def220 received from {username='slave'} at 127.28.17.5:42965
I20240501 13:59:44.528600 31134 tablet_copy_service.cc:433] P f3c5452a71ac4778977f154db2cb1659: ending tablet copy session b6318df9141f4124938a82e74ce5fdcd-20337968dab04bea87be1f0557def220 on tablet 20337968dab04bea87be1f0557def220 with peer b6318df9141f4124938a82e74ce5fdcd
I20240501 13:59:44.595049 391 raft_consensus.cc:1203] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.624732 32460 raft_consensus.cc:1203] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.669207 32327 raft_consensus.cc:1203] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.696674 391 raft_consensus.cc:1203] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.712872 32729 raft_consensus.cc:1203] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.730839 32729 raft_consensus.cc:1203] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.740116 32599 raft_consensus.cc:1203] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.938382 741 raft_consensus.cc:1050] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: attempting to promote NON_VOTER 3fc17cbb4edf4c11b1d6132ae4896997 to VOTER
I20240501 13:59:44.939308 741 consensus_queue.cc:237] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:44.941440 32327 raft_consensus.cc:1261] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Refusing update from remote peer f3c5452a71ac4778977f154db2cb1659: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:44.942193 739 consensus_queue.cc:1035] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:44.947037 741 raft_consensus.cc:2932] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } } }
I20240501 13:59:44.948060 32327 raft_consensus.cc:2932] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } } }
I20240501 13:59:44.952572 30968 catalog_manager.cc:5484] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index 4 to 5, 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:44.954190 32599 raft_consensus.cc:1203] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:44.981386 538 raft_consensus.cc:1050] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: attempting to promote NON_VOTER 92a0ab1d123c4dcba63e5e5064e059f6 to VOTER
I20240501 13:59:44.982000 538 consensus_queue.cc:237] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:44.984170 391 raft_consensus.cc:1261] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Refusing update from remote peer f3c5452a71ac4778977f154db2cb1659: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:44.984983 741 consensus_queue.cc:1035] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:44.989449 538 raft_consensus.cc:2932] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } } }
I20240501 13:59:44.991021 391 raft_consensus.cc:2932] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } } }
I20240501 13:59:44.995589 30966 catalog_manager.cc:5484] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index 4 to 5, 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.061233 741 raft_consensus.cc:1050] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: attempting to promote NON_VOTER 3fc17cbb4edf4c11b1d6132ae4896997 to VOTER
I20240501 13:59:45.061822 741 consensus_queue.cc:237] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:45.063598 32327 raft_consensus.cc:1261] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 LEARNER]: Refusing update from remote peer f3c5452a71ac4778977f154db2cb1659: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:45.064177 741 consensus_queue.cc:1035] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:45.066825 742 raft_consensus.cc:2932] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } } }
I20240501 13:59:45.067394 32327 raft_consensus.cc:2932] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } } }
I20240501 13:59:45.072221 30966 catalog_manager.cc:5484] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 reported cstate change: config changed from index 4 to 5, 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } } }
I20240501 13:59:45.122771 781 raft_consensus.cc:1050] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: attempting to promote NON_VOTER 9857990f442f4e7f98122344ebe75ffb to VOTER
I20240501 13:59:45.123600 781 consensus_queue.cc:237] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.125723 32729 raft_consensus.cc:1261] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Refusing update from remote peer 73ddc7b55a734843abb88e9f6d2be4b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:45.126472 780 consensus_queue.cc:1035] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:45.130167 781 raft_consensus.cc:2932] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } } }
I20240501 13:59:45.131038 32729 raft_consensus.cc:2932] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } } }
I20240501 13:59:45.135655 30968 catalog_manager.cc:5484] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index 4 to 5, 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.152987 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b71845f2e464353806c6306e4799456"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53976
I20240501 13:59:45.153366 31114 raft_consensus.cc:595] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.156931 525 raft_consensus.cc:1050] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: attempting to promote NON_VOTER d43e3c8c63554319b7bd3f1881889c32 to VOTER
I20240501 13:59:45.157590 525 consensus_queue.cc:237] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } }
I20240501 13:59:45.159840 32460 raft_consensus.cc:1261] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 1 LEARNER]: Refusing update from remote peer 73ddc7b55a734843abb88e9f6d2be4b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:45.160468 525 consensus_queue.cc:1035] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:45.163420 781 raft_consensus.cc:2932] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } } }
I20240501 13:59:45.164225 32460 raft_consensus.cc:2932] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } } }
I20240501 13:59:45.168591 30966 catalog_manager.cc:5484] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index 4 to 5, d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.181452 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52793644a7654afaa9b07340d3c02c1b"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53994
I20240501 13:59:45.181728 31114 raft_consensus.cc:595] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.205353 741 raft_consensus.cc:1050] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: attempting to promote NON_VOTER 92a0ab1d123c4dcba63e5e5064e059f6 to VOTER
I20240501 13:59:45.206072 741 consensus_queue.cc:237] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:45.207963 391 raft_consensus.cc:1261] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 LEARNER]: Refusing update from remote peer f3c5452a71ac4778977f154db2cb1659: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:45.208504 742 consensus_queue.cc:1035] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:45.211103 741 raft_consensus.cc:2932] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } } }
I20240501 13:59:45.211776 391 raft_consensus.cc:2932] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } } }
I20240501 13:59:45.216449 30966 catalog_manager.cc:5484] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index 4 to 5, 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.234290 31245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9834d3643f84f7dbedbca6bc836ed49"
dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32830
I20240501 13:59:45.234580 31245 raft_consensus.cc:595] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.252708 781 raft_consensus.cc:1050] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: attempting to promote NON_VOTER 9857990f442f4e7f98122344ebe75ffb to VOTER
I20240501 13:59:45.253427 781 consensus_queue.cc:237] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.253978 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226f3f612c1e4ca583c4b3a3e2a0c925"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54026
I20240501 13:59:45.254184 31114 raft_consensus.cc:595] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.255512 32729 raft_consensus.cc:1261] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 1 LEARNER]: Refusing update from remote peer 73ddc7b55a734843abb88e9f6d2be4b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:45.256214 780 consensus_queue.cc:1035] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:45.259075 525 raft_consensus.cc:2932] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } } }
I20240501 13:59:45.259682 32729 raft_consensus.cc:2932] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } } }
I20240501 13:59:45.264578 30968 catalog_manager.cc:5484] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index 4 to 5, 9857990f442f4e7f98122344ebe75ffb (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.278630 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4740a81166714d6bb90c11a657f0a8d9"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54056
I20240501 13:59:45.278865 31114 raft_consensus.cc:595] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.293907 781 raft_consensus.cc:1050] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: attempting to promote NON_VOTER b6318df9141f4124938a82e74ce5fdcd to VOTER
I20240501 13:59:45.294668 781 consensus_queue.cc:237] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.296895 32599 raft_consensus.cc:1261] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Refusing update from remote peer 73ddc7b55a734843abb88e9f6d2be4b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:45.297538 525 consensus_queue.cc:1035] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:45.297947 31245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2af886c958f649af819ee997d78babed"
dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32846
I20240501 13:59:45.298138 31245 raft_consensus.cc:595] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.307024 781 raft_consensus.cc:2932] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } } }
I20240501 13:59:45.307858 32599 raft_consensus.cc:2932] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } } }
I20240501 13:59:45.311326 30966 catalog_manager.cc:5484] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index 4 to 5, b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.337267 741 raft_consensus.cc:1050] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: attempting to promote NON_VOTER b6318df9141f4124938a82e74ce5fdcd to VOTER
I20240501 13:59:45.337955 741 consensus_queue.cc:237] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.339895 32599 raft_consensus.cc:1261] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 1 LEARNER]: Refusing update from remote peer f3c5452a71ac4778977f154db2cb1659: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:45.340413 538 consensus_queue.cc:1035] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:45.343310 741 raft_consensus.cc:2932] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } } }
I20240501 13:59:45.343973 32599 raft_consensus.cc:2932] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } } }
I20240501 13:59:45.348009 30968 catalog_manager.cc:5484] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index 4 to 5, b6318df9141f4124938a82e74ce5fdcd (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.401436 538 raft_consensus.cc:979] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: : Instructing follower 3fc17cbb4edf4c11b1d6132ae4896997 to start an election
I20240501 13:59:45.401618 538 raft_consensus.cc:1067] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Signalling peer 3fc17cbb4edf4c11b1d6132ae4896997 to start an election
I20240501 13:59:45.402642 32327 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4740a81166714d6bb90c11a657f0a8d9"
dest_uuid: "3fc17cbb4edf4c11b1d6132ae4896997"
from {username='slave'} at 127.28.17.1:57633
I20240501 13:59:45.402933 32327 raft_consensus.cc:482] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.403110 32327 raft_consensus.cc:3037] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.405146 32327 raft_consensus.cc:504] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:45.406437 32327 leader_election.cc:288] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [CANDIDATE]: Term 2 election: Requested vote from peers f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:45.407145 31114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4740a81166714d6bb90c11a657f0a8d9" candidate_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
I20240501 13:59:45.407459 31114 raft_consensus.cc:3032] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.407547 31114 raft_consensus.cc:729] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:45.407842 31114 consensus_queue.cc:260] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:45.408136 31114 raft_consensus.cc:3037] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.410221 31114 raft_consensus.cc:2454] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc17cbb4edf4c11b1d6132ae4896997 in term 2.
I20240501 13:59:45.410882 32261 leader_election.cc:302] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3fc17cbb4edf4c11b1d6132ae4896997, f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:45.411262 716 raft_consensus.cc:2781] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.411693 716 raft_consensus.cc:686] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 LEADER]: Becoming Leader. State: Replica: 3fc17cbb4edf4c11b1d6132ae4896997, State: Running, Role: LEADER
I20240501 13:59:45.412122 716 consensus_queue.cc:237] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:45.415593 30966 catalog_manager.cc:5484] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 reported cstate change: term changed from 1 to 2, leader changed from f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) to 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3). New cstate: current_term: 2 leader_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 13:59:45.426442 30960 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 4740a81166714d6bb90c11a657f0a8d9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3:37467) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 13:59:45.517283 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b71845f2e464353806c6306e4799456"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54068
I20240501 13:59:45.517558 31114 raft_consensus.cc:595] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.534435 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52793644a7654afaa9b07340d3c02c1b"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54088
I20240501 13:59:45.534704 31114 raft_consensus.cc:595] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.556704 31245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fe5276e21884c3eb35aa88d0849aa59"
dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32860
I20240501 13:59:45.556934 31245 raft_consensus.cc:595] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.573953 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20337968dab04bea87be1f0557def220"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54108
I20240501 13:59:45.574179 31114 raft_consensus.cc:595] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.576296 538 raft_consensus.cc:979] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: : Instructing follower 92a0ab1d123c4dcba63e5e5064e059f6 to start an election
I20240501 13:59:45.576467 538 raft_consensus.cc:1067] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Signalling peer 92a0ab1d123c4dcba63e5e5064e059f6 to start an election
I20240501 13:59:45.577288 391 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b71845f2e464353806c6306e4799456"
dest_uuid: "92a0ab1d123c4dcba63e5e5064e059f6"
from {username='slave'} at 127.28.17.1:46971
I20240501 13:59:45.577605 391 raft_consensus.cc:482] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.577827 391 raft_consensus.cc:3037] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.581211 391 raft_consensus.cc:504] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:45.582860 391 leader_election.cc:288] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [CANDIDATE]: Term 2 election: Requested vote from peers f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:45.583415 31114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b71845f2e464353806c6306e4799456" candidate_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
I20240501 13:59:45.583678 31114 raft_consensus.cc:3032] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.583763 31114 raft_consensus.cc:729] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:45.584031 31114 consensus_queue.cc:260] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:45.584288 31114 raft_consensus.cc:3037] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.586088 31114 raft_consensus.cc:2454] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a0ab1d123c4dcba63e5e5064e059f6 in term 2.
I20240501 13:59:45.586658 325 leader_election.cc:302] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 92a0ab1d123c4dcba63e5e5064e059f6, f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:45.587275 738 raft_consensus.cc:2781] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.587682 738 raft_consensus.cc:686] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 LEADER]: Becoming Leader. State: Replica: 92a0ab1d123c4dcba63e5e5064e059f6, State: Running, Role: LEADER
I20240501 13:59:45.588176 738 consensus_queue.cc:237] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:45.591961 30966 catalog_manager.cc:5484] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 reported cstate change: term changed from 1 to 2, leader changed from f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) to 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7). New cstate: current_term: 2 leader_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.597575 31245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea43ba0cf5e14827b80fbd3962b6bdbb"
dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32884
I20240501 13:59:45.597780 31245 raft_consensus.cc:595] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.609405 781 raft_consensus.cc:979] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: : Instructing follower d43e3c8c63554319b7bd3f1881889c32 to start an election
I20240501 13:59:45.609591 781 raft_consensus.cc:1067] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Signalling peer d43e3c8c63554319b7bd3f1881889c32 to start an election
I20240501 13:59:45.611210 32460 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2af886c958f649af819ee997d78babed"
dest_uuid: "d43e3c8c63554319b7bd3f1881889c32"
from {username='slave'} at 127.28.17.2:58317
I20240501 13:59:45.611549 32460 raft_consensus.cc:482] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.611680 32460 raft_consensus.cc:3037] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.614166 32460 raft_consensus.cc:504] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } }
I20240501 13:59:45.615392 32460 leader_election.cc:288] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [CANDIDATE]: Term 2 election: Requested vote from peers 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:45.616021 31245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af886c958f649af819ee997d78babed" candidate_uuid: "d43e3c8c63554319b7bd3f1881889c32" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
I20240501 13:59:45.616314 31245 raft_consensus.cc:3032] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.616438 31245 raft_consensus.cc:729] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:45.616794 31245 consensus_queue.cc:260] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } }
I20240501 13:59:45.617168 31245 raft_consensus.cc:3037] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.618021 31244 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9834d3643f84f7dbedbca6bc836ed49"
dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32912
I20240501 13:59:45.618253 31244 raft_consensus.cc:595] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.620608 31245 raft_consensus.cc:2454] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43e3c8c63554319b7bd3f1881889c32 in term 2.
I20240501 13:59:45.621308 32392 leader_election.cc:302] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7, d43e3c8c63554319b7bd3f1881889c32; no voters:
I20240501 13:59:45.621639 866 raft_consensus.cc:2781] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.622028 866 raft_consensus.cc:686] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 2 LEADER]: Becoming Leader. State: Replica: d43e3c8c63554319b7bd3f1881889c32, State: Running, Role: LEADER
I20240501 13:59:45.622345 866 consensus_queue.cc:237] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } }
I20240501 13:59:45.625612 30966 catalog_manager.cc:5484] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 reported cstate change: term changed from 1 to 2, leader changed from 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) to d43e3c8c63554319b7bd3f1881889c32 (127.28.17.4). New cstate: current_term: 2 leader_uuid: "d43e3c8c63554319b7bd3f1881889c32" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.639497 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "226f3f612c1e4ca583c4b3a3e2a0c925"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54126
I20240501 13:59:45.639724 31114 raft_consensus.cc:595] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.642483 742 raft_consensus.cc:979] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: : Instructing follower 92a0ab1d123c4dcba63e5e5064e059f6 to start an election
I20240501 13:59:45.642676 742 raft_consensus.cc:1067] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Signalling peer 92a0ab1d123c4dcba63e5e5064e059f6 to start an election
I20240501 13:59:45.643276 391 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "226f3f612c1e4ca583c4b3a3e2a0c925"
dest_uuid: "92a0ab1d123c4dcba63e5e5064e059f6"
from {username='slave'} at 127.28.17.1:46971
I20240501 13:59:45.643520 391 raft_consensus.cc:482] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.643630 391 raft_consensus.cc:3037] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.646238 391 raft_consensus.cc:504] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:45.646750 391 leader_election.cc:288] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [CANDIDATE]: Term 2 election: Requested vote from peers f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:45.647430 31114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226f3f612c1e4ca583c4b3a3e2a0c925" candidate_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
I20240501 13:59:45.647667 31114 raft_consensus.cc:3032] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.647751 31114 raft_consensus.cc:729] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:45.648032 31114 consensus_queue.cc:260] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:45.648640 31114 raft_consensus.cc:3037] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.650892 31114 raft_consensus.cc:2454] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a0ab1d123c4dcba63e5e5064e059f6 in term 2.
I20240501 13:59:45.651633 325 leader_election.cc:302] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 92a0ab1d123c4dcba63e5e5064e059f6, f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:45.651916 738 raft_consensus.cc:2781] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.652087 738 raft_consensus.cc:686] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 LEADER]: Becoming Leader. State: Replica: 92a0ab1d123c4dcba63e5e5064e059f6, State: Running, Role: LEADER
I20240501 13:59:45.652345 738 consensus_queue.cc:237] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:45.654848 30966 catalog_manager.cc:5484] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 reported cstate change: term changed from 1 to 2, leader changed from f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) to 92a0ab1d123c4dcba63e5e5064e059f6 (127.28.17.7). New cstate: current_term: 2 leader_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.658579 781 raft_consensus.cc:979] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: : Instructing follower 9857990f442f4e7f98122344ebe75ffb to start an election
I20240501 13:59:45.658757 781 raft_consensus.cc:1067] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Signalling peer 9857990f442f4e7f98122344ebe75ffb to start an election
I20240501 13:59:45.659458 32729 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9834d3643f84f7dbedbca6bc836ed49"
dest_uuid: "9857990f442f4e7f98122344ebe75ffb"
from {username='slave'} at 127.28.17.2:39569
I20240501 13:59:45.659714 32729 raft_consensus.cc:482] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.659875 32729 raft_consensus.cc:3037] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.662010 32729 raft_consensus.cc:504] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.663709 32729 leader_election.cc:288] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [CANDIDATE]: Term 2 election: Requested vote from peers 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:45.664302 31245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9834d3643f84f7dbedbca6bc836ed49" candidate_uuid: "9857990f442f4e7f98122344ebe75ffb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
I20240501 13:59:45.664494 31245 raft_consensus.cc:3032] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.664584 31245 raft_consensus.cc:729] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:45.664805 31245 consensus_queue.cc:260] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.665066 31245 raft_consensus.cc:3037] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.666957 31245 raft_consensus.cc:2454] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9857990f442f4e7f98122344ebe75ffb in term 2.
I20240501 13:59:45.667531 32663 leader_election.cc:302] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7, 9857990f442f4e7f98122344ebe75ffb; no voters:
I20240501 13:59:45.667883 744 raft_consensus.cc:2781] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.668256 744 raft_consensus.cc:686] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 2 LEADER]: Becoming Leader. State: Replica: 9857990f442f4e7f98122344ebe75ffb, State: Running, Role: LEADER
I20240501 13:59:45.668711 744 consensus_queue.cc:237] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.671588 30966 catalog_manager.cc:5484] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb reported cstate change: term changed from 1 to 2, leader changed from 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) to 9857990f442f4e7f98122344ebe75ffb (127.28.17.6). New cstate: current_term: 2 leader_uuid: "9857990f442f4e7f98122344ebe75ffb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.681097 741 raft_consensus.cc:979] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: : Instructing follower 3fc17cbb4edf4c11b1d6132ae4896997 to start an election
I20240501 13:59:45.681295 741 raft_consensus.cc:1067] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Signalling peer 3fc17cbb4edf4c11b1d6132ae4896997 to start an election
I20240501 13:59:45.682359 32327 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52793644a7654afaa9b07340d3c02c1b"
dest_uuid: "3fc17cbb4edf4c11b1d6132ae4896997"
from {username='slave'} at 127.28.17.1:57633
I20240501 13:59:45.682648 32327 raft_consensus.cc:482] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.682770 32327 raft_consensus.cc:3037] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.684072 781 raft_consensus.cc:979] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: : Instructing follower b6318df9141f4124938a82e74ce5fdcd to start an election
I20240501 13:59:45.684242 781 raft_consensus.cc:1067] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Signalling peer b6318df9141f4124938a82e74ce5fdcd to start an election
I20240501 13:59:45.684916 32599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea43ba0cf5e14827b80fbd3962b6bdbb"
dest_uuid: "b6318df9141f4124938a82e74ce5fdcd"
from {username='slave'} at 127.28.17.2:42533
I20240501 13:59:45.685016 32327 raft_consensus.cc:504] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:45.685158 32599 raft_consensus.cc:482] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.685349 32599 raft_consensus.cc:3037] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.685528 32327 leader_election.cc:288] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [CANDIDATE]: Term 2 election: Requested vote from peers f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:45.686877 31114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52793644a7654afaa9b07340d3c02c1b" candidate_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
I20240501 13:59:45.687074 31114 raft_consensus.cc:3032] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.687144 31114 raft_consensus.cc:729] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:45.687394 31114 consensus_queue.cc:260] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:45.687527 32599 raft_consensus.cc:504] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.687640 31114 raft_consensus.cc:3037] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.688731 32599 leader_election.cc:288] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [CANDIDATE]: Term 2 election: Requested vote from peers 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:45.689348 31245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea43ba0cf5e14827b80fbd3962b6bdbb" candidate_uuid: "b6318df9141f4124938a82e74ce5fdcd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
I20240501 13:59:45.689527 31245 raft_consensus.cc:3032] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.689630 31245 raft_consensus.cc:729] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:45.689877 31245 consensus_queue.cc:260] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.690176 31245 raft_consensus.cc:3037] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.690383 31114 raft_consensus.cc:2454] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc17cbb4edf4c11b1d6132ae4896997 in term 2.
I20240501 13:59:45.690797 32261 leader_election.cc:302] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3fc17cbb4edf4c11b1d6132ae4896997, f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:45.691082 716 raft_consensus.cc:2781] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.691228 716 raft_consensus.cc:686] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 LEADER]: Becoming Leader. State: Replica: 3fc17cbb4edf4c11b1d6132ae4896997, State: Running, Role: LEADER
I20240501 13:59:45.691468 716 consensus_queue.cc:237] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:45.692457 31245 raft_consensus.cc:2454] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6318df9141f4124938a82e74ce5fdcd in term 2.
I20240501 13:59:45.692965 32533 leader_election.cc:302] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7, b6318df9141f4124938a82e74ce5fdcd; no voters:
I20240501 13:59:45.693341 736 raft_consensus.cc:2781] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.693771 736 raft_consensus.cc:686] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 2 LEADER]: Becoming Leader. State: Replica: b6318df9141f4124938a82e74ce5fdcd, State: Running, Role: LEADER
I20240501 13:59:45.693665 30966 catalog_manager.cc:5484] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 reported cstate change: term changed from 1 to 2, leader changed from f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) to 3fc17cbb4edf4c11b1d6132ae4896997 (127.28.17.3). New cstate: current_term: 2 leader_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.694249 736 consensus_queue.cc:237] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.697837 30966 catalog_manager.cc:5484] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd reported cstate change: term changed from 1 to 2, leader changed from 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) to b6318df9141f4124938a82e74ce5fdcd (127.28.17.5). New cstate: current_term: 2 leader_uuid: "b6318df9141f4124938a82e74ce5fdcd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.762744 780 raft_consensus.cc:979] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: : Instructing follower 9857990f442f4e7f98122344ebe75ffb to start an election
I20240501 13:59:45.762907 780 raft_consensus.cc:1067] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Signalling peer 9857990f442f4e7f98122344ebe75ffb to start an election
I20240501 13:59:45.763625 32729 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6fe5276e21884c3eb35aa88d0849aa59"
dest_uuid: "9857990f442f4e7f98122344ebe75ffb"
from {username='slave'} at 127.28.17.2:39569
I20240501 13:59:45.763815 32729 raft_consensus.cc:482] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.763914 32729 raft_consensus.cc:3037] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.766146 32729 raft_consensus.cc:504] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.766659 32729 leader_election.cc:288] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [CANDIDATE]: Term 2 election: Requested vote from peers 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:45.767190 31245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe5276e21884c3eb35aa88d0849aa59" candidate_uuid: "9857990f442f4e7f98122344ebe75ffb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
I20240501 13:59:45.767367 31245 raft_consensus.cc:3032] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.767465 31245 raft_consensus.cc:729] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:45.767696 31245 consensus_queue.cc:260] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.767997 31245 raft_consensus.cc:3037] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.770257 31245 raft_consensus.cc:2454] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9857990f442f4e7f98122344ebe75ffb in term 2.
I20240501 13:59:45.770704 32663 leader_election.cc:302] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ddc7b55a734843abb88e9f6d2be4b7, 9857990f442f4e7f98122344ebe75ffb; no voters:
I20240501 13:59:45.771013 744 raft_consensus.cc:2781] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.771162 744 raft_consensus.cc:686] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 2 LEADER]: Becoming Leader. State: Replica: 9857990f442f4e7f98122344ebe75ffb, State: Running, Role: LEADER
I20240501 13:59:45.771412 744 consensus_queue.cc:237] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:45.773443 30966 catalog_manager.cc:5484] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb reported cstate change: term changed from 1 to 2, leader changed from 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) to 9857990f442f4e7f98122344ebe75ffb (127.28.17.6). New cstate: current_term: 2 leader_uuid: "9857990f442f4e7f98122344ebe75ffb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.907826 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20337968dab04bea87be1f0557def220"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54140
I20240501 13:59:45.908044 31114 raft_consensus.cc:595] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:45.925849 742 raft_consensus.cc:979] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: : Instructing follower b6318df9141f4124938a82e74ce5fdcd to start an election
I20240501 13:59:45.926005 742 raft_consensus.cc:1067] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Signalling peer b6318df9141f4124938a82e74ce5fdcd to start an election
I20240501 13:59:45.926856 32599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20337968dab04bea87be1f0557def220"
dest_uuid: "b6318df9141f4124938a82e74ce5fdcd"
from {username='slave'} at 127.28.17.1:34805
I20240501 13:59:45.927085 32599 raft_consensus.cc:482] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:45.927206 32599 raft_consensus.cc:3037] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.929065 32599 raft_consensus.cc:504] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.930043 32599 leader_election.cc:288] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [CANDIDATE]: Term 2 election: Requested vote from peers f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:45.930752 31114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20337968dab04bea87be1f0557def220" candidate_uuid: "b6318df9141f4124938a82e74ce5fdcd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
I20240501 13:59:45.930979 31114 raft_consensus.cc:3032] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:45.931048 31114 raft_consensus.cc:729] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:45.931324 31114 consensus_queue.cc:260] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.931594 31114 raft_consensus.cc:3037] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:45.933495 31114 raft_consensus.cc:2454] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6318df9141f4124938a82e74ce5fdcd in term 2.
I20240501 13:59:45.933893 32533 leader_election.cc:302] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b6318df9141f4124938a82e74ce5fdcd, f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:45.934185 736 raft_consensus.cc:2781] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:45.934429 736 raft_consensus.cc:686] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 2 LEADER]: Becoming Leader. State: Replica: b6318df9141f4124938a82e74ce5fdcd, State: Running, Role: LEADER
I20240501 13:59:45.934737 736 consensus_queue.cc:237] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:45.936857 30966 catalog_manager.cc:5484] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd reported cstate change: term changed from 1 to 2, leader changed from f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) to b6318df9141f4124938a82e74ce5fdcd (127.28.17.5). New cstate: current_term: 2 leader_uuid: "b6318df9141f4124938a82e74ce5fdcd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:45.969398 31114 raft_consensus.cc:1261] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Refusing update from remote peer 3fc17cbb4edf4c11b1d6132ae4896997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:45.970360 935 consensus_queue.cc:1035] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.058672 31114 raft_consensus.cc:1261] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Refusing update from remote peer 92a0ab1d123c4dcba63e5e5064e059f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.059355 983 consensus_queue.cc:1035] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.061887 32327 consensus_queue.cc:237] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:46.062866 935 raft_consensus.cc:2932] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } } }
I20240501 13:59:46.066864 30960 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4740a81166714d6bb90c11a657f0a8d9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.067818 30968 catalog_manager.cc:5484] T 4740a81166714d6bb90c11a657f0a8d9 P 3fc17cbb4edf4c11b1d6132ae4896997 reported cstate change: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New cstate: current_term: 2 leader_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.071218 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4740a81166714d6bb90c11a657f0a8d9 with delete_type TABLET_DATA_TOMBSTONED (TS f3c5452a71ac4778977f154db2cb1659 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38402
I20240501 13:59:46.071923 1080 tablet_replica.cc:330] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: stopping tablet replica
I20240501 13:59:46.072320 1080 raft_consensus.cc:2229] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.072574 1080 raft_consensus.cc:2258] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.073912 1080 ts_tablet_manager.cc:1875] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.078575 1080 ts_tablet_manager.cc:1888] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.078737 1080 log.cc:1196] T 4740a81166714d6bb90c11a657f0a8d9 P f3c5452a71ac4778977f154db2cb1659: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/4740a81166714d6bb90c11a657f0a8d9
I20240501 13:59:46.079525 30958 catalog_manager.cc:4830] TS f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235): tablet 4740a81166714d6bb90c11a657f0a8d9 (table rebalance_test_table_0 [id=fbf4be62827b4639bb92e73b579e499a]) successfully deleted
I20240501 13:59:46.169394 31245 raft_consensus.cc:1261] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Refusing update from remote peer d43e3c8c63554319b7bd3f1881889c32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.170302 1086 consensus_queue.cc:1035] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.184406 31245 raft_consensus.cc:1261] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Refusing update from remote peer 9857990f442f4e7f98122344ebe75ffb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.185101 744 consensus_queue.cc:1035] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.190670 31114 raft_consensus.cc:1261] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Refusing update from remote peer 92a0ab1d123c4dcba63e5e5064e059f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.191269 738 consensus_queue.cc:1035] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.256071 31245 raft_consensus.cc:1261] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Refusing update from remote peer b6318df9141f4124938a82e74ce5fdcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.256651 391 consensus_queue.cc:237] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:46.256956 736 consensus_queue.cc:1035] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.257658 983 raft_consensus.cc:2932] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } } }
I20240501 13:59:46.262954 30960 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b71845f2e464353806c6306e4799456 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.264487 391 consensus_queue.cc:237] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } }
I20240501 13:59:46.264545 30966 catalog_manager.cc:5484] T 6b71845f2e464353806c6306e4799456 P 92a0ab1d123c4dcba63e5e5064e059f6 reported cstate change: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New cstate: current_term: 2 leader_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.265376 983 raft_consensus.cc:2932] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } } }
I20240501 13:59:46.268043 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b71845f2e464353806c6306e4799456 with delete_type TABLET_DATA_TOMBSTONED (TS f3c5452a71ac4778977f154db2cb1659 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38402
I20240501 13:59:46.268319 30960 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 226f3f612c1e4ca583c4b3a3e2a0c925 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.268395 1080 tablet_replica.cc:330] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: stopping tablet replica
I20240501 13:59:46.268581 1080 raft_consensus.cc:2229] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.268764 1080 raft_consensus.cc:2258] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.269670 1080 ts_tablet_manager.cc:1875] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.269555 30966 catalog_manager.cc:5484] T 226f3f612c1e4ca583c4b3a3e2a0c925 P 92a0ab1d123c4dcba63e5e5064e059f6 reported cstate change: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New cstate: current_term: 2 leader_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "92a0ab1d123c4dcba63e5e5064e059f6" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 45721 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.272945 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet 226f3f612c1e4ca583c4b3a3e2a0c925 with delete_type TABLET_DATA_TOMBSTONED (TS f3c5452a71ac4778977f154db2cb1659 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38402
I20240501 13:59:46.275708 1080 ts_tablet_manager.cc:1888] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.275826 1080 log.cc:1196] T 6b71845f2e464353806c6306e4799456 P f3c5452a71ac4778977f154db2cb1659: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/6b71845f2e464353806c6306e4799456
I20240501 13:59:46.276473 1080 tablet_replica.cc:330] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: stopping tablet replica
I20240501 13:59:46.276515 30958 catalog_manager.cc:4830] TS f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235): tablet 6b71845f2e464353806c6306e4799456 (table rebalance_test_table_0 [id=fbf4be62827b4639bb92e73b579e499a]) successfully deleted
I20240501 13:59:46.276765 1080 raft_consensus.cc:2229] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.276966 1080 raft_consensus.cc:2258] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.278262 1080 ts_tablet_manager.cc:1875] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.283766 1080 ts_tablet_manager.cc:1888] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.283994 1080 log.cc:1196] T 226f3f612c1e4ca583c4b3a3e2a0c925 P f3c5452a71ac4778977f154db2cb1659: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/226f3f612c1e4ca583c4b3a3e2a0c925
I20240501 13:59:46.284703 30958 catalog_manager.cc:4830] TS f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235): tablet 226f3f612c1e4ca583c4b3a3e2a0c925 (table rebalance_test_table_2 [id=27639d7577424922b8eeeef214bcd4d0]) successfully deleted
I20240501 13:59:46.285177 31114 raft_consensus.cc:1261] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Refusing update from remote peer 3fc17cbb4edf4c11b1d6132ae4896997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.285779 716 consensus_queue.cc:1035] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.288311 32460 consensus_queue.cc:237] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } }
I20240501 13:59:46.290344 1093 raft_consensus.cc:2932] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } } }
I20240501 13:59:46.293648 30959 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2af886c958f649af819ee997d78babed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.295075 30968 catalog_manager.cc:5484] T 2af886c958f649af819ee997d78babed P d43e3c8c63554319b7bd3f1881889c32 reported cstate change: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New cstate: current_term: 2 leader_uuid: "d43e3c8c63554319b7bd3f1881889c32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d43e3c8c63554319b7bd3f1881889c32" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 34759 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.298774 31224 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af886c958f649af819ee997d78babed with delete_type TABLET_DATA_TOMBSTONED (TS 73ddc7b55a734843abb88e9f6d2be4b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38680
I20240501 13:59:46.299639 1154 tablet_replica.cc:330] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: stopping tablet replica
I20240501 13:59:46.300134 1154 raft_consensus.cc:2229] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.300410 1154 raft_consensus.cc:2258] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.302675 1154 ts_tablet_manager.cc:1875] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.309247 1154 ts_tablet_manager.cc:1888] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.309404 1154 log.cc:1196] T 2af886c958f649af819ee997d78babed P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/2af886c958f649af819ee997d78babed
I20240501 13:59:46.310269 30958 catalog_manager.cc:4830] TS 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045): tablet 2af886c958f649af819ee997d78babed (table rebalance_test_table_1 [id=4fb2daa2611c4ac88c321041f5f617e1]) successfully deleted
I20240501 13:59:46.313205 32729 consensus_queue.cc:237] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:46.314131 1103 raft_consensus.cc:2932] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } } }
I20240501 13:59:46.317273 30958 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9834d3643f84f7dbedbca6bc836ed49 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.318578 30968 catalog_manager.cc:5484] T f9834d3643f84f7dbedbca6bc836ed49 P 9857990f442f4e7f98122344ebe75ffb reported cstate change: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New cstate: current_term: 2 leader_uuid: "9857990f442f4e7f98122344ebe75ffb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.322021 31224 tablet_service.cc:1514] Processing DeleteTablet for tablet f9834d3643f84f7dbedbca6bc836ed49 with delete_type TABLET_DATA_TOMBSTONED (TS 73ddc7b55a734843abb88e9f6d2be4b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38680
I20240501 13:59:46.322386 1154 tablet_replica.cc:330] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: stopping tablet replica
I20240501 13:59:46.322638 1154 raft_consensus.cc:2229] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.323343 1154 raft_consensus.cc:2258] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.324360 1154 ts_tablet_manager.cc:1875] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.325614 32599 consensus_queue.cc:237] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:46.326591 1141 raft_consensus.cc:2932] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } } }
I20240501 13:59:46.329030 1154 ts_tablet_manager.cc:1888] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.329159 1154 log.cc:1196] T f9834d3643f84f7dbedbca6bc836ed49 P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9834d3643f84f7dbedbca6bc836ed49
I20240501 13:59:46.329571 30960 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea43ba0cf5e14827b80fbd3962b6bdbb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.330633 30958 catalog_manager.cc:4830] TS 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045): tablet f9834d3643f84f7dbedbca6bc836ed49 (table rebalance_test_table_4 [id=c758a9c63300499caf401bd3f77940bb]) successfully deleted
I20240501 13:59:46.330899 30968 catalog_manager.cc:5484] T ea43ba0cf5e14827b80fbd3962b6bdbb P b6318df9141f4124938a82e74ce5fdcd reported cstate change: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New cstate: current_term: 2 leader_uuid: "b6318df9141f4124938a82e74ce5fdcd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.334363 31224 tablet_service.cc:1514] Processing DeleteTablet for tablet ea43ba0cf5e14827b80fbd3962b6bdbb with delete_type TABLET_DATA_TOMBSTONED (TS 73ddc7b55a734843abb88e9f6d2be4b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38680
I20240501 13:59:46.334751 1154 tablet_replica.cc:330] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: stopping tablet replica
I20240501 13:59:46.334926 1154 raft_consensus.cc:2229] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.335099 1154 raft_consensus.cc:2258] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.335904 1154 ts_tablet_manager.cc:1875] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.341934 1154 ts_tablet_manager.cc:1888] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.342051 1154 log.cc:1196] T ea43ba0cf5e14827b80fbd3962b6bdbb P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/ea43ba0cf5e14827b80fbd3962b6bdbb
I20240501 13:59:46.342759 30958 catalog_manager.cc:4830] TS 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045): tablet ea43ba0cf5e14827b80fbd3962b6bdbb (table rebalance_test_table_3 [id=a3ffcfa40e2b4ecdbdf18c2a3555b352]) successfully deleted
I20240501 13:59:46.345135 32327 consensus_queue.cc:237] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } }
I20240501 13:59:46.345996 935 raft_consensus.cc:2932] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } } }
I20240501 13:59:46.349826 30960 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52793644a7654afaa9b07340d3c02c1b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.351176 30968 catalog_manager.cc:5484] T 52793644a7654afaa9b07340d3c02c1b P 3fc17cbb4edf4c11b1d6132ae4896997 reported cstate change: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New cstate: current_term: 2 leader_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fc17cbb4edf4c11b1d6132ae4896997" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.354880 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet 52793644a7654afaa9b07340d3c02c1b with delete_type TABLET_DATA_TOMBSTONED (TS f3c5452a71ac4778977f154db2cb1659 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38402
I20240501 13:59:46.355250 1080 tablet_replica.cc:330] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: stopping tablet replica
I20240501 13:59:46.355438 1080 raft_consensus.cc:2229] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.355649 1080 raft_consensus.cc:2258] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.356482 1080 ts_tablet_manager.cc:1875] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.362003 1080 ts_tablet_manager.cc:1888] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.362139 1080 log.cc:1196] T 52793644a7654afaa9b07340d3c02c1b P f3c5452a71ac4778977f154db2cb1659: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/52793644a7654afaa9b07340d3c02c1b
I20240501 13:59:46.362277 31245 raft_consensus.cc:1261] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Refusing update from remote peer 9857990f442f4e7f98122344ebe75ffb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.362903 30958 catalog_manager.cc:4830] TS f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235): tablet 52793644a7654afaa9b07340d3c02c1b (table rebalance_test_table_2 [id=27639d7577424922b8eeeef214bcd4d0]) successfully deleted
I20240501 13:59:46.363025 744 consensus_queue.cc:1035] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.408829 32729 consensus_queue.cc:237] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } }
I20240501 13:59:46.410023 1022 raft_consensus.cc:2932] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } } }
I20240501 13:59:46.413039 30958 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6fe5276e21884c3eb35aa88d0849aa59 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.414239 30966 catalog_manager.cc:5484] T 6fe5276e21884c3eb35aa88d0849aa59 P 9857990f442f4e7f98122344ebe75ffb reported cstate change: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New cstate: current_term: 2 leader_uuid: "9857990f442f4e7f98122344ebe75ffb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9857990f442f4e7f98122344ebe75ffb" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 42705 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.417819 31224 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe5276e21884c3eb35aa88d0849aa59 with delete_type TABLET_DATA_TOMBSTONED (TS 73ddc7b55a734843abb88e9f6d2be4b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38680
I20240501 13:59:46.418169 1154 tablet_replica.cc:330] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: stopping tablet replica
I20240501 13:59:46.418401 1154 raft_consensus.cc:2229] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.418640 1154 raft_consensus.cc:2258] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.419721 1154 ts_tablet_manager.cc:1875] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.425493 1154 ts_tablet_manager.cc:1888] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.425639 1154 log.cc:1196] T 6fe5276e21884c3eb35aa88d0849aa59 P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/6fe5276e21884c3eb35aa88d0849aa59
I20240501 13:59:46.426270 30958 catalog_manager.cc:4830] TS 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045): tablet 6fe5276e21884c3eb35aa88d0849aa59 (table rebalance_test_table_3 [id=a3ffcfa40e2b4ecdbdf18c2a3555b352]) successfully deleted
I20240501 13:59:46.499975 31114 raft_consensus.cc:1261] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Refusing update from remote peer b6318df9141f4124938a82e74ce5fdcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:46.500650 736 consensus_queue.cc:1035] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:46.559829 32599 consensus_queue.cc:237] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } }
I20240501 13:59:46.560711 1141 raft_consensus.cc:2932] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } } }
I20240501 13:59:46.563449 30960 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20337968dab04bea87be1f0557def220 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:46.564677 30966 catalog_manager.cc:5484] T 20337968dab04bea87be1f0557def220 P b6318df9141f4124938a82e74ce5fdcd reported cstate change: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New cstate: current_term: 2 leader_uuid: "b6318df9141f4124938a82e74ce5fdcd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b6318df9141f4124938a82e74ce5fdcd" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 46587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:46.568188 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet 20337968dab04bea87be1f0557def220 with delete_type TABLET_DATA_TOMBSTONED (TS f3c5452a71ac4778977f154db2cb1659 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38402
I20240501 13:59:46.568555 1080 tablet_replica.cc:330] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: stopping tablet replica
I20240501 13:59:46.568713 1080 raft_consensus.cc:2229] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:46.568871 1080 raft_consensus.cc:2258] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:46.569615 1080 ts_tablet_manager.cc:1875] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:46.573887 1080 ts_tablet_manager.cc:1888] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:46.573998 1080 log.cc:1196] T 20337968dab04bea87be1f0557def220 P f3c5452a71ac4778977f154db2cb1659: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/20337968dab04bea87be1f0557def220
I20240501 13:59:46.574735 30958 catalog_manager.cc:4830] TS f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235): tablet 20337968dab04bea87be1f0557def220 (table rebalance_test_table_4 [id=c758a9c63300499caf401bd3f77940bb]) successfully deleted
I20240501 13:59:46.762610 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:46.768770 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:46.775511 31224 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:46.784538 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:46.785203 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:46.786733 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:46.842733 31094 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:46.991364 31224 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:46.997468 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.001956 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.003806 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.017253 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.020119 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.035713 31094 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:47.731122 31224 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:47.737756 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.740974 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.756469 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.788938 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.801714 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:47.811728 31094 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:48.005847 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.011327 31224 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:48.021113 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.022884 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.043764 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.066097 31094 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:48.081300 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.324147 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.339752 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.340180 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.344014 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:48.363983 31094 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:48.373544 31224 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:48.375594 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
26e34a0aa8454cc9810f40277fc27268 | 127.28.17.62:46735 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.17.20:42559 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
3fc17cbb4edf4c11b1d6132ae4896997 | 127.28.17.3:37467 | HEALTHY | <none> | 2 | 0
73ddc7b55a734843abb88e9f6d2be4b7 | 127.28.17.2:36045 | HEALTHY | <none> | 3 | 0
92a0ab1d123c4dcba63e5e5064e059f6 | 127.28.17.7:45721 | HEALTHY | <none> | 2 | 0
9857990f442f4e7f98122344ebe75ffb | 127.28.17.6:42705 | HEALTHY | <none> | 2 | 0
b6318df9141f4124938a82e74ce5fdcd | 127.28.17.5:46587 | HEALTHY | <none> | 2 | 0
d43e3c8c63554319b7bd3f1881889c32 | 127.28.17.4:34759 | HEALTHY | <none> | 2 | 0
f3c5452a71ac4778977f154db2cb1659 | 127.28.17.1:44235 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.28.17.1 | experimental | 127.28.17.1:44235
local_ip_for_outbound_sockets | 127.28.17.2 | experimental | 127.28.17.2:36045
local_ip_for_outbound_sockets | 127.28.17.3 | experimental | 127.28.17.3:37467
local_ip_for_outbound_sockets | 127.28.17.4 | experimental | 127.28.17.4:34759
local_ip_for_outbound_sockets | 127.28.17.5 | experimental | 127.28.17.5:46587
local_ip_for_outbound_sockets | 127.28.17.6 | experimental | 127.28.17.6:42705
local_ip_for_outbound_sockets | 127.28.17.7 | experimental | 127.28.17.7:45721
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.17.1:44235
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.17.2:36045
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.17.3:37467
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.17.4:34759
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.17.5:46587
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.17.6:42705
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.28.17.7:45721
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.17.20:42559 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 13:59:48.456622 28740 log_verifier.cc:126] Checking tablet 20337968dab04bea87be1f0557def220
I20240501 13:59:48.472990 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 20337968dab04bea87be1f0557def220
I20240501 13:59:48.473148 28740 log_verifier.cc:126] Checking tablet 226f3f612c1e4ca583c4b3a3e2a0c925
I20240501 13:59:48.485630 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 226f3f612c1e4ca583c4b3a3e2a0c925
I20240501 13:59:48.485826 28740 log_verifier.cc:126] Checking tablet 2af886c958f649af819ee997d78babed
I20240501 13:59:48.504804 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2af886c958f649af819ee997d78babed
I20240501 13:59:48.504972 28740 log_verifier.cc:126] Checking tablet 2d6e706bde6441d7b9fdbfe69731da5a
I20240501 13:59:48.517735 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2d6e706bde6441d7b9fdbfe69731da5a
I20240501 13:59:48.517858 28740 log_verifier.cc:126] Checking tablet 4740a81166714d6bb90c11a657f0a8d9
I20240501 13:59:48.530270 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 4740a81166714d6bb90c11a657f0a8d9
I20240501 13:59:48.530386 28740 log_verifier.cc:126] Checking tablet 52793644a7654afaa9b07340d3c02c1b
I20240501 13:59:48.543048 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 52793644a7654afaa9b07340d3c02c1b
I20240501 13:59:48.543221 28740 log_verifier.cc:126] Checking tablet 6b71845f2e464353806c6306e4799456
I20240501 13:59:48.555500 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6b71845f2e464353806c6306e4799456
I20240501 13:59:48.555619 28740 log_verifier.cc:126] Checking tablet 6fe5276e21884c3eb35aa88d0849aa59
I20240501 13:59:48.568948 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6fe5276e21884c3eb35aa88d0849aa59
I20240501 13:59:48.569068 28740 log_verifier.cc:126] Checking tablet 71040610692d4cd3babce1e103b2bd08
I20240501 13:59:48.581804 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 71040610692d4cd3babce1e103b2bd08
I20240501 13:59:48.581919 28740 log_verifier.cc:126] Checking tablet 77be73948b1248fa8e6dd0de4dd65467
I20240501 13:59:48.594588 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 77be73948b1248fa8e6dd0de4dd65467
I20240501 13:59:48.594697 28740 log_verifier.cc:126] Checking tablet 81f2b5656f784e8caa4ade4c76228899
I20240501 13:59:48.603308 28740 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 81f2b5656f784e8caa4ade4c76228899
I20240501 13:59:48.603425 28740 log_verifier.cc:126] Checking tablet 97c6a2dd8f154b61810ae1aa0ee90b49
I20240501 13:59:48.615852 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 97c6a2dd8f154b61810ae1aa0ee90b49
I20240501 13:59:48.616034 28740 log_verifier.cc:126] Checking tablet c77d9c76031c47aa994f1bf89139cb1d
I20240501 13:59:48.629022 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c77d9c76031c47aa994f1bf89139cb1d
I20240501 13:59:48.629117 28740 log_verifier.cc:126] Checking tablet e642f36e29f842ac80dddcfda574623b
I20240501 13:59:48.641983 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e642f36e29f842ac80dddcfda574623b
I20240501 13:59:48.642092 28740 log_verifier.cc:126] Checking tablet ea43ba0cf5e14827b80fbd3962b6bdbb
I20240501 13:59:48.655602 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ea43ba0cf5e14827b80fbd3962b6bdbb
I20240501 13:59:48.655725 28740 log_verifier.cc:126] Checking tablet f9834d3643f84f7dbedbca6bc836ed49
I20240501 13:59:48.669445 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f9834d3643f84f7dbedbca6bc836ed49
I20240501 13:59:48.672391 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.8:0
--local_ip_for_outbound_sockets=127.28.17.8
--webserver_interface=127.28.17.8
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--builtin_ntp_servers=127.28.17.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:49.026217 1402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:49.026575 1402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:49.026685 1402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:49.034670 1402 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:49.034793 1402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:49.034910 1402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.8
I20240501 13:59:49.045152 1402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.17.8
--webserver_port=0
--tserver_master_addrs=127.28.17.62:46735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.8
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:49.046480 1402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:49.048401 1402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:49.062218 1407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:49.062831 1410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:49.063243 1408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:49.063792 1402 server_base.cc:1034] running on GCE node
I20240501 13:59:49.064453 1402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:49.065461 1402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:49.066627 1402 hybrid_clock.cc:648] HybridClock initialized: now 1714571989066575 us; error 47 us; skew 500 ppm
I20240501 13:59:49.066960 1402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:49.069314 1402 webserver.cc:460] Webserver started at http://127.28.17.8:39081/ using document root <none> and password file <none>
I20240501 13:59:49.070053 1402 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:49.070166 1402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:49.070518 1402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:49.073155 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0ab1797c43b44025a9b0f91dde5d0eb3"
format_stamp: "Formatted at 2024-05-01 13:59:49 on dist-test-slave-dx8c"
I20240501 13:59:49.073936 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0ab1797c43b44025a9b0f91dde5d0eb3"
format_stamp: "Formatted at 2024-05-01 13:59:49 on dist-test-slave-dx8c"
I20240501 13:59:49.079797 1402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240501 13:59:49.083416 1416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:49.084787 1402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:49.084954 1402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0ab1797c43b44025a9b0f91dde5d0eb3"
format_stamp: "Formatted at 2024-05-01 13:59:49 on dist-test-slave-dx8c"
I20240501 13:59:49.085150 1402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:49.109557 1402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:49.110430 1402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:49.110738 1402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:49.111703 1402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:49.113178 1402 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:49.113294 1402 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:49.113400 1402 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:49.113481 1402 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:49.145710 1402 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.8:43205
I20240501 13:59:49.145754 1528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.8:43205 every 8 connection(s)
I20240501 13:59:49.147132 1402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 13:59:49.154559 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 1402
I20240501 13:59:49.154789 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 13:59:49.161096 1529 heartbeater.cc:347] Connected to a master server at 127.28.17.62:46735
I20240501 13:59:49.161465 1529 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:49.162525 1529 heartbeater.cc:511] Master 127.28.17.62:46735 requested a full tablet report, sending...
I20240501 13:59:49.164220 30967 ts_manager.cc:194] Registered new tserver with Master: 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8:43205)
I20240501 13:59:49.165340 30967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.8:50555
I20240501 13:59:49.651206 31224 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.654163 1464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:49.669479 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.674836 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.685662 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.693672 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.718906 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.738953 31094 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.914381 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.924361 1464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 13:59:49.932781 31224 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:49.936235 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.946544 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.952291 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.967397 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:49.969946 31094 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 13:59:50.076272 31114 consensus_queue.cc:237] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } }
I20240501 13:59:50.078336 1623 raft_consensus.cc:2932] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } } }
I20240501 13:59:50.084798 30968 catalog_manager.cc:5484] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index -1 to 4, NON_VOTER 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:50.089444 31049 consensus_peers.cc:485] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 -> Peer 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8:43205): Couldn't send request to peer 0ab1797c43b44025a9b0f91dde5d0eb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c77d9c76031c47aa994f1bf89139cb1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:50.100791 31245 consensus_queue.cc:237] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } }
I20240501 13:59:50.106997 1636 raft_consensus.cc:2932] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } } }
I20240501 13:59:50.114677 30970 catalog_manager.cc:5484] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 13:59:50.119410 31180 consensus_peers.cc:485] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 -> Peer 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8:43205): Couldn't send request to peer 0ab1797c43b44025a9b0f91dde5d0eb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77be73948b1248fa8e6dd0de4dd65467. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:50.168818 1529 heartbeater.cc:503] Master 127.28.17.62:46735 was elected leader, sending a full tablet report...
I20240501 13:59:50.551935 1658 ts_tablet_manager.cc:902] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: Initiating tablet copy from peer f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:50.554399 1658 tablet_copy_client.cc:308] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:44235
I20240501 13:59:50.560497 31134 tablet_copy_service.cc:140] P f3c5452a71ac4778977f154db2cb1659: Received BeginTabletCopySession request for tablet c77d9c76031c47aa994f1bf89139cb1d from peer 0ab1797c43b44025a9b0f91dde5d0eb3 ({username='slave'} at 127.28.17.8:58749)
I20240501 13:59:50.560693 31134 tablet_copy_service.cc:161] P f3c5452a71ac4778977f154db2cb1659: Beginning new tablet copy session on tablet c77d9c76031c47aa994f1bf89139cb1d from peer 0ab1797c43b44025a9b0f91dde5d0eb3 at {username='slave'} at 127.28.17.8:58749: session id = 0ab1797c43b44025a9b0f91dde5d0eb3-c77d9c76031c47aa994f1bf89139cb1d
I20240501 13:59:50.563159 31134 tablet_copy_source_session.cc:205] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:50.568168 1658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77d9c76031c47aa994f1bf89139cb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:50.579572 1658 tablet_copy_client.cc:783] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:50.579927 1658 tablet_copy_client.cc:647] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:50.581876 1658 tablet_copy_client.cc:523] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:50.585511 1658 tablet_bootstrap.cc:492] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: Bootstrap starting.
I20240501 13:59:50.594767 1658 log.cc:824] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:50.607056 1675 ts_tablet_manager.cc:902] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: Initiating tablet copy from peer 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:50.608650 1675 tablet_copy_client.cc:308] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:36045
I20240501 13:59:50.614820 31265 tablet_copy_service.cc:140] P 73ddc7b55a734843abb88e9f6d2be4b7: Received BeginTabletCopySession request for tablet 77be73948b1248fa8e6dd0de4dd65467 from peer 0ab1797c43b44025a9b0f91dde5d0eb3 ({username='slave'} at 127.28.17.8:34005)
I20240501 13:59:50.615011 31265 tablet_copy_service.cc:161] P 73ddc7b55a734843abb88e9f6d2be4b7: Beginning new tablet copy session on tablet 77be73948b1248fa8e6dd0de4dd65467 from peer 0ab1797c43b44025a9b0f91dde5d0eb3 at {username='slave'} at 127.28.17.8:34005: session id = 0ab1797c43b44025a9b0f91dde5d0eb3-77be73948b1248fa8e6dd0de4dd65467
I20240501 13:59:50.617769 31265 tablet_copy_source_session.cc:205] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:50.619253 1675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77be73948b1248fa8e6dd0de4dd65467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:50.624261 1675 tablet_copy_client.cc:783] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:50.624521 1675 tablet_copy_client.cc:647] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:50.626490 1675 tablet_copy_client.cc:523] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:50.629005 1675 tablet_bootstrap.cc:492] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: Bootstrap starting.
I20240501 13:59:50.631343 1658 tablet_bootstrap.cc:492] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:50.631879 1658 tablet_bootstrap.cc:492] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: Bootstrap complete.
I20240501 13:59:50.632337 1658 ts_tablet_manager.cc:1367] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: Time spent bootstrapping tablet: real 0.047s user 0.042s sys 0.006s
I20240501 13:59:50.637790 1658 raft_consensus.cc:348] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } }
I20240501 13:59:50.638233 1658 raft_consensus.cc:729] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ab1797c43b44025a9b0f91dde5d0eb3, State: Initialized, Role: LEARNER
I20240501 13:59:50.638794 1658 consensus_queue.cc:260] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } }
I20240501 13:59:50.640422 1658 ts_tablet_manager.cc:1398] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 13:59:50.641392 31134 tablet_copy_service.cc:342] P f3c5452a71ac4778977f154db2cb1659: Request end of tablet copy session 0ab1797c43b44025a9b0f91dde5d0eb3-c77d9c76031c47aa994f1bf89139cb1d received from {username='slave'} at 127.28.17.8:58749
I20240501 13:59:50.641541 31134 tablet_copy_service.cc:433] P f3c5452a71ac4778977f154db2cb1659: ending tablet copy session 0ab1797c43b44025a9b0f91dde5d0eb3-c77d9c76031c47aa994f1bf89139cb1d on tablet c77d9c76031c47aa994f1bf89139cb1d with peer 0ab1797c43b44025a9b0f91dde5d0eb3
I20240501 13:59:50.660935 1675 tablet_bootstrap.cc:492] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:50.661391 1675 tablet_bootstrap.cc:492] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: Bootstrap complete.
I20240501 13:59:50.661656 1675 ts_tablet_manager.cc:1367] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20240501 13:59:50.662581 1675 raft_consensus.cc:348] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } }
I20240501 13:59:50.662768 1675 raft_consensus.cc:729] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ab1797c43b44025a9b0f91dde5d0eb3, State: Initialized, Role: LEARNER
I20240501 13:59:50.662946 1675 consensus_queue.cc:260] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: true } }
I20240501 13:59:50.663340 1675 ts_tablet_manager.cc:1398] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 13:59:50.664126 31265 tablet_copy_service.cc:342] P 73ddc7b55a734843abb88e9f6d2be4b7: Request end of tablet copy session 0ab1797c43b44025a9b0f91dde5d0eb3-77be73948b1248fa8e6dd0de4dd65467 received from {username='slave'} at 127.28.17.8:34005
I20240501 13:59:50.664286 31265 tablet_copy_service.cc:433] P 73ddc7b55a734843abb88e9f6d2be4b7: ending tablet copy session 0ab1797c43b44025a9b0f91dde5d0eb3-77be73948b1248fa8e6dd0de4dd65467 on tablet 77be73948b1248fa8e6dd0de4dd65467 with peer 0ab1797c43b44025a9b0f91dde5d0eb3
I20240501 13:59:50.948484 1484 raft_consensus.cc:1203] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:51.143129 1484 raft_consensus.cc:1203] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 13:59:51.555131 1728 raft_consensus.cc:1050] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: attempting to promote NON_VOTER 0ab1797c43b44025a9b0f91dde5d0eb3 to VOTER
I20240501 13:59:51.555771 1728 consensus_queue.cc:237] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:51.557776 1484 raft_consensus.cc:1261] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Refusing update from remote peer f3c5452a71ac4778977f154db2cb1659: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:51.558403 1727 consensus_queue.cc:1035] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:51.561869 1728 raft_consensus.cc:2932] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } } }
I20240501 13:59:51.562749 1484 raft_consensus.cc:2932] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } } }
I20240501 13:59:51.567008 30968 catalog_manager.cc:5484] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 reported cstate change: config changed from index 4 to 5, 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f3c5452a71ac4778977f154db2cb1659" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:51.694265 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c77d9c76031c47aa994f1bf89139cb1d"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50120
I20240501 13:59:51.694552 31114 raft_consensus.cc:595] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:51.733888 1752 raft_consensus.cc:1050] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: attempting to promote NON_VOTER 0ab1797c43b44025a9b0f91dde5d0eb3 to VOTER
I20240501 13:59:51.734413 1752 consensus_queue.cc:237] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:51.736223 1484 raft_consensus.cc:1261] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 LEARNER]: Refusing update from remote peer 73ddc7b55a734843abb88e9f6d2be4b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 13:59:51.736723 1752 consensus_queue.cc:1035] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:59:51.739300 1751 raft_consensus.cc:2932] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } } }
I20240501 13:59:51.739899 1484 raft_consensus.cc:2932] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } } }
I20240501 13:59:51.743568 30970 catalog_manager.cc:5484] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 reported cstate change: config changed from index 4 to 5, 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:51.917236 31245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77be73948b1248fa8e6dd0de4dd65467"
dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47220
I20240501 13:59:51.917474 31245 raft_consensus.cc:595] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:51.934559 31114 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c77d9c76031c47aa994f1bf89139cb1d"
dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50138
I20240501 13:59:51.934783 31114 raft_consensus.cc:595] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:52.064080 1731 raft_consensus.cc:979] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: : Instructing follower 0ab1797c43b44025a9b0f91dde5d0eb3 to start an election
I20240501 13:59:52.064252 1731 raft_consensus.cc:1067] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Signalling peer 0ab1797c43b44025a9b0f91dde5d0eb3 to start an election
I20240501 13:59:52.065019 1484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c77d9c76031c47aa994f1bf89139cb1d"
dest_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3"
from {username='slave'} at 127.28.17.1:40597
I20240501 13:59:52.065291 1484 raft_consensus.cc:482] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:52.065466 1484 raft_consensus.cc:3037] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:52.067435 1484 raft_consensus.cc:504] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.068676 1484 leader_election.cc:288] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [CANDIDATE]: Term 2 election: Requested vote from peers f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235)
I20240501 13:59:52.069234 31114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77d9c76031c47aa994f1bf89139cb1d" candidate_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f3c5452a71ac4778977f154db2cb1659"
I20240501 13:59:52.069417 31114 raft_consensus.cc:3032] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:52.069504 31114 raft_consensus.cc:729] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3c5452a71ac4778977f154db2cb1659, State: Running, Role: LEADER
I20240501 13:59:52.069772 31114 consensus_queue.cc:260] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.070029 31114 raft_consensus.cc:3037] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:52.071883 31114 raft_consensus.cc:2454] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ab1797c43b44025a9b0f91dde5d0eb3 in term 2.
I20240501 13:59:52.072491 1418 leader_election.cc:302] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ab1797c43b44025a9b0f91dde5d0eb3, f3c5452a71ac4778977f154db2cb1659; no voters:
I20240501 13:59:52.072885 1681 raft_consensus.cc:2781] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:52.073259 1681 raft_consensus.cc:686] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 LEADER]: Becoming Leader. State: Replica: 0ab1797c43b44025a9b0f91dde5d0eb3, State: Running, Role: LEADER
I20240501 13:59:52.073746 1681 consensus_queue.cc:237] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.077085 30968 catalog_manager.cc:5484] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 reported cstate change: term changed from 1 to 2, leader changed from f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) to 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8). New cstate: current_term: 2 leader_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 13:59:52.086414 30959 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet c77d9c76031c47aa994f1bf89139cb1d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8:43205) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 13:59:52.154753 31245 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77be73948b1248fa8e6dd0de4dd65467"
dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47236
I20240501 13:59:52.155022 31245 raft_consensus.cc:595] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 13:59:52.197079 1751 raft_consensus.cc:979] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: : Instructing follower 0ab1797c43b44025a9b0f91dde5d0eb3 to start an election
I20240501 13:59:52.197269 1751 raft_consensus.cc:1067] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Signalling peer 0ab1797c43b44025a9b0f91dde5d0eb3 to start an election
I20240501 13:59:52.197943 1484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77be73948b1248fa8e6dd0de4dd65467"
dest_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3"
from {username='slave'} at 127.28.17.2:41401
I20240501 13:59:52.198168 1484 raft_consensus.cc:482] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:52.198273 1484 raft_consensus.cc:3037] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:52.200141 1484 raft_consensus.cc:504] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.200737 1484 leader_election.cc:288] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [CANDIDATE]: Term 2 election: Requested vote from peers 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045)
I20240501 13:59:52.201287 31245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be73948b1248fa8e6dd0de4dd65467" candidate_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "73ddc7b55a734843abb88e9f6d2be4b7"
I20240501 13:59:52.201488 31245 raft_consensus.cc:3032] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:52.201567 31245 raft_consensus.cc:729] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ddc7b55a734843abb88e9f6d2be4b7, State: Running, Role: LEADER
I20240501 13:59:52.201823 31245 consensus_queue.cc:260] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.202071 31245 raft_consensus.cc:3037] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:52.204012 31245 raft_consensus.cc:2454] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ab1797c43b44025a9b0f91dde5d0eb3 in term 2.
I20240501 13:59:52.204457 1418 leader_election.cc:302] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ab1797c43b44025a9b0f91dde5d0eb3, 73ddc7b55a734843abb88e9f6d2be4b7; no voters:
I20240501 13:59:52.204731 1681 raft_consensus.cc:2781] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:52.204886 1681 raft_consensus.cc:686] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 LEADER]: Becoming Leader. State: Replica: 0ab1797c43b44025a9b0f91dde5d0eb3, State: Running, Role: LEADER
I20240501 13:59:52.205123 1681 consensus_queue.cc:237] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.207113 30968 catalog_manager.cc:5484] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 reported cstate change: term changed from 1 to 2, leader changed from 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) to 0ab1797c43b44025a9b0f91dde5d0eb3 (127.28.17.8). New cstate: current_term: 2 leader_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:52.696888 31114 raft_consensus.cc:1261] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Refusing update from remote peer 0ab1797c43b44025a9b0f91dde5d0eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:52.697737 1681 consensus_queue.cc:1035] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c5452a71ac4778977f154db2cb1659" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:52.710108 1484 consensus_queue.cc:237] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.710942 1776 raft_consensus.cc:2932] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } } }
I20240501 13:59:52.713552 30959 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c77d9c76031c47aa994f1bf89139cb1d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:52.714746 30970 catalog_manager.cc:5484] T c77d9c76031c47aa994f1bf89139cb1d P 0ab1797c43b44025a9b0f91dde5d0eb3 reported cstate change: config changed from index 5 to 7, VOTER f3c5452a71ac4778977f154db2cb1659 (127.28.17.1) evicted. New cstate: current_term: 2 leader_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:52.718822 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet c77d9c76031c47aa994f1bf89139cb1d with delete_type TABLET_DATA_TOMBSTONED (TS f3c5452a71ac4778977f154db2cb1659 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38402
I20240501 13:59:52.719385 1821 tablet_replica.cc:330] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: stopping tablet replica
I20240501 13:59:52.719568 1821 raft_consensus.cc:2229] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:52.719749 1821 raft_consensus.cc:2258] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:52.721055 1821 ts_tablet_manager.cc:1875] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:52.725409 1821 ts_tablet_manager.cc:1888] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:52.725522 1821 log.cc:1196] T c77d9c76031c47aa994f1bf89139cb1d P f3c5452a71ac4778977f154db2cb1659: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/c77d9c76031c47aa994f1bf89139cb1d
I20240501 13:59:52.726321 30958 catalog_manager.cc:4830] TS f3c5452a71ac4778977f154db2cb1659 (127.28.17.1:44235): tablet c77d9c76031c47aa994f1bf89139cb1d (table rebalance_test_table_3 [id=a3ffcfa40e2b4ecdbdf18c2a3555b352]) successfully deleted
I20240501 13:59:52.785858 31245 raft_consensus.cc:1261] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Refusing update from remote peer 0ab1797c43b44025a9b0f91dde5d0eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 13:59:52.786612 1776 consensus_queue.cc:1035] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ddc7b55a734843abb88e9f6d2be4b7" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 36045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 13:59:52.842621 1484 consensus_queue.cc:237] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } }
I20240501 13:59:52.843547 1681 raft_consensus.cc:2932] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } } }
I20240501 13:59:52.846275 30959 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77be73948b1248fa8e6dd0de4dd65467 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 13:59:52.847635 30970 catalog_manager.cc:5484] T 77be73948b1248fa8e6dd0de4dd65467 P 0ab1797c43b44025a9b0f91dde5d0eb3 reported cstate change: config changed from index 5 to 7, VOTER 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2) evicted. New cstate: current_term: 2 leader_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab1797c43b44025a9b0f91dde5d0eb3" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 43205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:52.850883 31224 tablet_service.cc:1514] Processing DeleteTablet for tablet 77be73948b1248fa8e6dd0de4dd65467 with delete_type TABLET_DATA_TOMBSTONED (TS 73ddc7b55a734843abb88e9f6d2be4b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38680
I20240501 13:59:52.851549 1840 tablet_replica.cc:330] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: stopping tablet replica
I20240501 13:59:52.851724 1840 raft_consensus.cc:2229] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:52.851904 1840 raft_consensus.cc:2258] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:52.853297 1840 ts_tablet_manager.cc:1875] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:52.857643 1840 ts_tablet_manager.cc:1888] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 13:59:52.857779 1840 log.cc:1196] T 77be73948b1248fa8e6dd0de4dd65467 P 73ddc7b55a734843abb88e9f6d2be4b7: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/77be73948b1248fa8e6dd0de4dd65467
I20240501 13:59:52.858582 30958 catalog_manager.cc:4830] TS 73ddc7b55a734843abb88e9f6d2be4b7 (127.28.17.2:36045): tablet 77be73948b1248fa8e6dd0de4dd65467 (table rebalance_test_table_0 [id=fbf4be62827b4639bb92e73b579e499a]) successfully deleted
I20240501 13:59:53.156833 31224 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.166357 1464 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.188482 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.205495 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.213768 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.225342 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.254365 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.294993 31094 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.425487 1464 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.436810 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.438930 31224 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.456091 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.456532 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.484546 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.486732 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.491730 31094 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.773021 32307 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.773897 1464 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.787088 371 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.803732 32579 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.804778 31224 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.815624 32709 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.818789 32439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 13:59:53.841732 31094 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
26e34a0aa8454cc9810f40277fc27268 | 127.28.17.62:46735 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.17.20:42559 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
0ab1797c43b44025a9b0f91dde5d0eb3 | 127.28.17.8:43205 | HEALTHY | <none> | 2 | 0
3fc17cbb4edf4c11b1d6132ae4896997 | 127.28.17.3:37467 | HEALTHY | <none> | 2 | 0
73ddc7b55a734843abb88e9f6d2be4b7 | 127.28.17.2:36045 | HEALTHY | <none> | 2 | 0
92a0ab1d123c4dcba63e5e5064e059f6 | 127.28.17.7:45721 | HEALTHY | <none> | 2 | 0
9857990f442f4e7f98122344ebe75ffb | 127.28.17.6:42705 | HEALTHY | <none> | 2 | 0
b6318df9141f4124938a82e74ce5fdcd | 127.28.17.5:46587 | HEALTHY | <none> | 2 | 0
d43e3c8c63554319b7bd3f1881889c32 | 127.28.17.4:34759 | HEALTHY | <none> | 2 | 0
f3c5452a71ac4778977f154db2cb1659 | 127.28.17.1:44235 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.28.17.1 | experimental | 127.28.17.1:44235
local_ip_for_outbound_sockets | 127.28.17.2 | experimental | 127.28.17.2:36045
local_ip_for_outbound_sockets | 127.28.17.3 | experimental | 127.28.17.3:37467
local_ip_for_outbound_sockets | 127.28.17.4 | experimental | 127.28.17.4:34759
local_ip_for_outbound_sockets | 127.28.17.5 | experimental | 127.28.17.5:46587
local_ip_for_outbound_sockets | 127.28.17.6 | experimental | 127.28.17.6:42705
local_ip_for_outbound_sockets | 127.28.17.7 | experimental | 127.28.17.7:45721
local_ip_for_outbound_sockets | 127.28.17.8 | experimental | 127.28.17.8:43205
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.17.1:44235
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.17.2:36045
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.17.3:37467
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.17.4:34759
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.17.5:46587
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.17.6:42705
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.28.17.7:45721
server_dump_info_path | /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.28.17.8:43205
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.17.20:42559 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 13:59:53.908933 28740 log_verifier.cc:126] Checking tablet 20337968dab04bea87be1f0557def220
I20240501 13:59:53.919183 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 20337968dab04bea87be1f0557def220
I20240501 13:59:53.919289 28740 log_verifier.cc:126] Checking tablet 226f3f612c1e4ca583c4b3a3e2a0c925
I20240501 13:59:53.929067 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 226f3f612c1e4ca583c4b3a3e2a0c925
I20240501 13:59:53.929163 28740 log_verifier.cc:126] Checking tablet 2af886c958f649af819ee997d78babed
I20240501 13:59:53.939395 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2af886c958f649af819ee997d78babed
I20240501 13:59:53.939494 28740 log_verifier.cc:126] Checking tablet 2d6e706bde6441d7b9fdbfe69731da5a
I20240501 13:59:53.948632 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2d6e706bde6441d7b9fdbfe69731da5a
I20240501 13:59:53.948719 28740 log_verifier.cc:126] Checking tablet 4740a81166714d6bb90c11a657f0a8d9
I20240501 13:59:53.958603 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 4740a81166714d6bb90c11a657f0a8d9
I20240501 13:59:53.958694 28740 log_verifier.cc:126] Checking tablet 52793644a7654afaa9b07340d3c02c1b
I20240501 13:59:53.968469 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 52793644a7654afaa9b07340d3c02c1b
I20240501 13:59:53.968564 28740 log_verifier.cc:126] Checking tablet 6b71845f2e464353806c6306e4799456
I20240501 13:59:53.978215 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6b71845f2e464353806c6306e4799456
I20240501 13:59:53.978318 28740 log_verifier.cc:126] Checking tablet 6fe5276e21884c3eb35aa88d0849aa59
I20240501 13:59:53.988107 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6fe5276e21884c3eb35aa88d0849aa59
I20240501 13:59:53.988197 28740 log_verifier.cc:126] Checking tablet 71040610692d4cd3babce1e103b2bd08
I20240501 13:59:53.997311 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 71040610692d4cd3babce1e103b2bd08
I20240501 13:59:53.997421 28740 log_verifier.cc:126] Checking tablet 77be73948b1248fa8e6dd0de4dd65467
I20240501 13:59:54.009717 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 77be73948b1248fa8e6dd0de4dd65467
I20240501 13:59:54.009822 28740 log_verifier.cc:126] Checking tablet 81f2b5656f784e8caa4ade4c76228899
I20240501 13:59:54.014701 28740 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 81f2b5656f784e8caa4ade4c76228899
I20240501 13:59:54.014786 28740 log_verifier.cc:126] Checking tablet 97c6a2dd8f154b61810ae1aa0ee90b49
I20240501 13:59:54.023772 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 97c6a2dd8f154b61810ae1aa0ee90b49
I20240501 13:59:54.023859 28740 log_verifier.cc:126] Checking tablet c77d9c76031c47aa994f1bf89139cb1d
I20240501 13:59:54.036911 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c77d9c76031c47aa994f1bf89139cb1d
I20240501 13:59:54.037014 28740 log_verifier.cc:126] Checking tablet e642f36e29f842ac80dddcfda574623b
I20240501 13:59:54.046602 28740 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e642f36e29f842ac80dddcfda574623b
I20240501 13:59:54.046685 28740 log_verifier.cc:126] Checking tablet ea43ba0cf5e14827b80fbd3962b6bdbb
I20240501 13:59:54.056689 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ea43ba0cf5e14827b80fbd3962b6bdbb
I20240501 13:59:54.056792 28740 log_verifier.cc:126] Checking tablet f9834d3643f84f7dbedbca6bc836ed49
I20240501 13:59:54.066246 28740 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f9834d3643f84f7dbedbca6bc836ed49
I20240501 13:59:54.067139 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 31032
I20240501 13:59:54.068686 31153 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:54.205734 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 31032
I20240501 13:59:54.226203 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 31163
I20240501 13:59:54.227272 31284 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:54.371227 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 31163
I20240501 13:59:54.391782 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 32245
I20240501 13:59:54.397805 32367 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:54.511126 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 32245
I20240501 13:59:54.528343 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 32376
I20240501 13:59:54.534260 32503 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:54.645992 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 32376
I20240501 13:59:54.662201 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 32513
I20240501 13:59:54.667903 32639 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:54.788969 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 32513
I20240501 13:59:54.805591 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 32647
I20240501 13:59:54.811504 300 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:54.925287 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 32647
I20240501 13:59:54.942322 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 309
I20240501 13:59:54.948302 431 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:55.062212 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 309
I20240501 13:59:55.080468 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 1402
I20240501 13:59:55.086155 1523 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:55.186455 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 1402
I20240501 13:59:55.201300 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 30942
I20240501 13:59:55.202428 31003 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:55.350302 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 30942
2024-05-01T13:59:55Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (27372 ms)
[----------] 1 test from RebalanceParamTest (27372 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240501 13:59:55.410272 28740 test_util.cc:268] Using random seed: -423989431
I20240501 13:59:55.412191 28740 ts_itest-base.cc:112] Starting cluster with:
I20240501 13:59:55.412262 28740 ts_itest-base.cc:113] --------------
I20240501 13:59:55.412323 28740 ts_itest-base.cc:114] 8 tablet servers
I20240501 13:59:55.412366 28740 ts_itest-base.cc:115] 3 replicas per TS
I20240501 13:59:55.412425 28740 ts_itest-base.cc:116] --------------
2024-05-01T13:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T13:59:55Z Disabled control of system clock
I20240501 13:59:55.439935 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:33799
--webserver_interface=127.28.17.62
--webserver_port=0
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.17.62:33799
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 13:59:55.766851 1975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:55.767179 1975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:55.767269 1975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:55.774755 1975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:55.774888 1975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 13:59:55.774940 1975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:55.774995 1975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 13:59:55.775040 1975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 13:59:55.784332 1975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.17.62:33799
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.17.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:55.785610 1975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:55.787392 1975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:55.798156 1980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:55.798554 1983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:55.799135 1981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:55.799759 1975 server_base.cc:1034] running on GCE node
I20240501 13:59:55.800601 1975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:55.802098 1975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:55.803298 1975 hybrid_clock.cc:648] HybridClock initialized: now 1714571995803231 us; error 62 us; skew 500 ppm
I20240501 13:59:55.803658 1975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:55.805924 1975 webserver.cc:460] Webserver started at http://127.28.17.62:34211/ using document root <none> and password file <none>
I20240501 13:59:55.806744 1975 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:55.806861 1975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:55.807169 1975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:55.809924 1975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "09c4cadf5e9843128fcc82f0a1415ab1"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-dx8c"
I20240501 13:59:55.810843 1975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "09c4cadf5e9843128fcc82f0a1415ab1"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-dx8c"
I20240501 13:59:55.816150 1975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 13:59:55.819607 1989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:55.820999 1975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:55.821172 1975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
uuid: "09c4cadf5e9843128fcc82f0a1415ab1"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-dx8c"
I20240501 13:59:55.821395 1975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:55.838548 1975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:55.839443 1975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:55.839702 1975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:55.855602 1975 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.62:33799
I20240501 13:59:55.855608 2040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.62:33799 every 8 connection(s)
I20240501 13:59:55.856998 1975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 13:59:55.858795 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 1975
I20240501 13:59:55.858955 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 13:59:55.861065 2041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:55.874261 2041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Bootstrap starting.
I20240501 13:59:55.877609 2041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:55.878656 2041 log.cc:824] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:55.881253 2041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: No bootstrap required, opened a new log
I20240501 13:59:55.887224 2041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 33799 } }
I20240501 13:59:55.887588 2041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:55.887666 2041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c4cadf5e9843128fcc82f0a1415ab1, State: Initialized, Role: FOLLOWER
I20240501 13:59:55.888236 2041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 33799 } }
I20240501 13:59:55.888419 2041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:55.888521 2041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:55.888648 2041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:55.890667 2041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 33799 } }
I20240501 13:59:55.891127 2041 leader_election.cc:302] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09c4cadf5e9843128fcc82f0a1415ab1; no voters:
I20240501 13:59:55.891726 2041 leader_election.cc:288] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:55.891852 2046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:55.892515 2046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [term 1 LEADER]: Becoming Leader. State: Replica: 09c4cadf5e9843128fcc82f0a1415ab1, State: Running, Role: LEADER
I20240501 13:59:55.893011 2046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 33799 } }
I20240501 13:59:55.893677 2041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:55.895685 2048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09c4cadf5e9843128fcc82f0a1415ab1. Latest consensus state: current_term: 1 leader_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 33799 } } }
I20240501 13:59:55.896060 2048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:55.898720 2047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09c4cadf5e9843128fcc82f0a1415ab1" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 33799 } } }
I20240501 13:59:55.898979 2047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:55.900096 2055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:55.904500 2055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:55.913205 2055 catalog_manager.cc:1347] Generated new cluster ID: e90a9aaa616e48dbb31e8ecf1f6610f6
I20240501 13:59:55.913625 2055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:55.933637 2055 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:55.934960 2055 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:55.949052 2055 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Generated new TSK 0
I20240501 13:59:55.949936 2055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:55.966642 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.1:0
--local_ip_for_outbound_sockets=127.28.17.1
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 13:59:56.313227 2065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:56.313534 2065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:56.313665 2065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:56.321328 2065 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:56.321462 2065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:56.321621 2065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.1
I20240501 13:59:56.331270 2065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.1
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:56.332607 2065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:56.334306 2065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:56.346354 2071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:56.346810 2073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:56.347323 2070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:56.347879 2065 server_base.cc:1034] running on GCE node
I20240501 13:59:56.348497 2065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:56.349416 2065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:56.350584 2065 hybrid_clock.cc:648] HybridClock initialized: now 1714571996350533 us; error 47 us; skew 500 ppm
I20240501 13:59:56.350896 2065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:56.353099 2065 webserver.cc:460] Webserver started at http://127.28.17.1:36983/ using document root <none> and password file <none>
I20240501 13:59:56.353853 2065 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:56.353966 2065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:56.354264 2065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:56.356966 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d1a414d381f44829a2510cb76d481887"
format_stamp: "Formatted at 2024-05-01 13:59:56 on dist-test-slave-dx8c"
I20240501 13:59:56.357777 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d1a414d381f44829a2510cb76d481887"
format_stamp: "Formatted at 2024-05-01 13:59:56 on dist-test-slave-dx8c"
I20240501 13:59:56.363077 2065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:56.366485 2079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:56.367916 2065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:56.368074 2065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d1a414d381f44829a2510cb76d481887"
format_stamp: "Formatted at 2024-05-01 13:59:56 on dist-test-slave-dx8c"
I20240501 13:59:56.368264 2065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:56.394986 2065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:56.395875 2065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:56.396148 2065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:56.397125 2065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:56.398788 2065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:56.398890 2065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:56.398999 2065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:56.399068 2065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:56.428546 2065 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.1:41309
I20240501 13:59:56.428617 2191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.1:41309 every 8 connection(s)
I20240501 13:59:56.429894 2065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 13:59:56.439327 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2065
I20240501 13:59:56.439574 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 13:59:56.442864 2192 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:56.443253 2192 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:56.443208 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.2:0
--local_ip_for_outbound_sockets=127.28.17.2
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:56.444312 2192 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:56.446749 2006 ts_manager.cc:194] Registered new tserver with Master: d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 13:59:56.448452 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.1:34703
W20240501 13:59:56.770990 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:56.771314 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:56.771453 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:56.779168 2196 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:56.779438 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:56.779629 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.2
I20240501 13:59:56.789014 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.2
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:56.790366 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:56.792217 2196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:56.804479 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:56.804931 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:56.805600 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:56.806248 2196 server_base.cc:1034] running on GCE node
I20240501 13:59:56.806927 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:56.807916 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:56.809152 2196 hybrid_clock.cc:648] HybridClock initialized: now 1714571996809086 us; error 52 us; skew 500 ppm
I20240501 13:59:56.809517 2196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:56.811935 2196 webserver.cc:460] Webserver started at http://127.28.17.2:34063/ using document root <none> and password file <none>
I20240501 13:59:56.812726 2196 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:56.812839 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:56.813148 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:56.816059 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
format_stamp: "Formatted at 2024-05-01 13:59:56 on dist-test-slave-dx8c"
I20240501 13:59:56.816948 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
format_stamp: "Formatted at 2024-05-01 13:59:56 on dist-test-slave-dx8c"
I20240501 13:59:56.822531 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 13:59:56.825992 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:56.827517 2196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:56.827690 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
format_stamp: "Formatted at 2024-05-01 13:59:56 on dist-test-slave-dx8c"
I20240501 13:59:56.827898 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:56.846189 2196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:56.847124 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:56.847419 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:56.848392 2196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:56.849958 2196 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:56.850052 2196 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:56.850162 2196 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:56.850229 2196 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:56.880673 2196 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.2:46301
I20240501 13:59:56.880717 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.2:46301 every 8 connection(s)
I20240501 13:59:56.881945 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 13:59:56.883003 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2196
I20240501 13:59:56.883167 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 13:59:56.886895 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.3:0
--local_ip_for_outbound_sockets=127.28.17.3
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:56.894982 2323 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:56.895330 2323 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:56.896353 2323 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:56.897910 2006 ts_manager.cc:194] Registered new tserver with Master: b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 13:59:56.898732 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.2:35403
W20240501 13:59:57.220386 2326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:57.220679 2326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:57.220801 2326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:57.228421 2326 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:57.228551 2326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:57.228710 2326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.3
I20240501 13:59:57.237773 2326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.3
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:57.239136 2326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:57.240906 2326 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:57.253201 2331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:57.253820 2332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:57.254307 2334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:57.254827 2326 server_base.cc:1034] running on GCE node
I20240501 13:59:57.255499 2326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:57.256554 2326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:57.257784 2326 hybrid_clock.cc:648] HybridClock initialized: now 1714571997257717 us; error 63 us; skew 500 ppm
I20240501 13:59:57.258121 2326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:57.265787 2326 webserver.cc:460] Webserver started at http://127.28.17.3:38527/ using document root <none> and password file <none>
I20240501 13:59:57.266777 2326 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:57.266914 2326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:57.267247 2326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:57.270002 2326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
format_stamp: "Formatted at 2024-05-01 13:59:57 on dist-test-slave-dx8c"
I20240501 13:59:57.270869 2326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
format_stamp: "Formatted at 2024-05-01 13:59:57 on dist-test-slave-dx8c"
I20240501 13:59:57.276408 2326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20240501 13:59:57.280226 2340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:57.281706 2326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:57.281888 2326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
format_stamp: "Formatted at 2024-05-01 13:59:57 on dist-test-slave-dx8c"
I20240501 13:59:57.282099 2326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:57.302661 2326 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:57.303591 2326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:57.303889 2326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:57.304898 2326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:57.306466 2326 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:57.306583 2326 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240501 13:59:57.306700 2326 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:57.306777 2326 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:57.336621 2326 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.3:33537
I20240501 13:59:57.336679 2452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.3:33537 every 8 connection(s)
I20240501 13:59:57.337922 2326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 13:59:57.348136 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2326
I20240501 13:59:57.348361 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 13:59:57.350805 2453 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:57.351155 2453 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:57.351766 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.4:0
--local_ip_for_outbound_sockets=127.28.17.4
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:57.352226 2453 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:57.353899 2006 ts_manager.cc:194] Registered new tserver with Master: a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 13:59:57.354931 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.3:37951
I20240501 13:59:57.451663 2192 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
W20240501 13:59:57.689280 2456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:57.689574 2456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:57.689666 2456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:57.697280 2456 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:57.697410 2456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:57.697525 2456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.4
I20240501 13:59:57.706630 2456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.4
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:57.707896 2456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:57.709584 2456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:57.721932 2462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:57.722431 2461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:57.722772 2464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:57.723475 2456 server_base.cc:1034] running on GCE node
I20240501 13:59:57.724085 2456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:57.725076 2456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:57.726285 2456 hybrid_clock.cc:648] HybridClock initialized: now 1714571997726239 us; error 38 us; skew 500 ppm
I20240501 13:59:57.726673 2456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:57.728960 2456 webserver.cc:460] Webserver started at http://127.28.17.4:36537/ using document root <none> and password file <none>
I20240501 13:59:57.729724 2456 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:57.729837 2456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:57.730144 2456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:57.732980 2456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "745ac12076e84effa5bde1ecbd6a32ec"
format_stamp: "Formatted at 2024-05-01 13:59:57 on dist-test-slave-dx8c"
I20240501 13:59:57.733783 2456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "745ac12076e84effa5bde1ecbd6a32ec"
format_stamp: "Formatted at 2024-05-01 13:59:57 on dist-test-slave-dx8c"
I20240501 13:59:57.739179 2456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 13:59:57.742786 2470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:57.744318 2456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:57.744478 2456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "745ac12076e84effa5bde1ecbd6a32ec"
format_stamp: "Formatted at 2024-05-01 13:59:57 on dist-test-slave-dx8c"
I20240501 13:59:57.744678 2456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:57.763275 2456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:57.764144 2456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:57.764410 2456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:57.765372 2456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:57.766925 2456 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:57.767015 2456 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:57.767112 2456 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:57.767174 2456 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:57.797132 2456 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.4:36117
I20240501 13:59:57.797219 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.4:36117 every 8 connection(s)
I20240501 13:59:57.798569 2456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 13:59:57.802513 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2456
I20240501 13:59:57.802690 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 13:59:57.805850 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:0
--local_ip_for_outbound_sockets=127.28.17.5
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:57.812490 2583 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:57.812852 2583 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:57.813948 2583 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:57.815603 2006 ts_manager.cc:194] Registered new tserver with Master: 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 13:59:57.816623 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.4:33693
I20240501 13:59:57.901924 2323 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
W20240501 13:59:58.141299 2586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:58.141633 2586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:58.141772 2586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:58.149425 2586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:58.149585 2586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:58.149717 2586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 13:59:58.158793 2586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:58.160074 2586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:58.161852 2586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:58.174275 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:58.174546 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:58.175101 2594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:58.175829 2586 server_base.cc:1034] running on GCE node
I20240501 13:59:58.176473 2586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:58.177444 2586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:58.178624 2586 hybrid_clock.cc:648] HybridClock initialized: now 1714571998178570 us; error 48 us; skew 500 ppm
I20240501 13:59:58.178962 2586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:58.181308 2586 webserver.cc:460] Webserver started at http://127.28.17.5:42069/ using document root <none> and password file <none>
I20240501 13:59:58.182089 2586 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:58.182211 2586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:58.182574 2586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:58.185240 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 13:59:58.186082 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 13:59:58.191570 2586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 13:59:58.195230 2600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:58.196663 2586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:58.196831 2586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 13:59:58.197031 2586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:58.212966 2586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:58.213825 2586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:58.214092 2586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:58.215060 2586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:58.216506 2586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:58.216600 2586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:58.216710 2586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:58.216778 2586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:58.247033 2586 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:40059
I20240501 13:59:58.247099 2712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:40059 every 8 connection(s)
I20240501 13:59:58.248415 2586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 13:59:58.257761 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2586
I20240501 13:59:58.257972 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 13:59:58.261068 2713 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:58.261277 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.6:0
--local_ip_for_outbound_sockets=127.28.17.6
--webserver_interface=127.28.17.6
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:58.261437 2713 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:58.262517 2713 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:58.264130 2006 ts_manager.cc:194] Registered new tserver with Master: f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 13:59:58.264917 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:47841
I20240501 13:59:58.358026 2453 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
W20240501 13:59:58.590476 2716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:58.590816 2716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:58.590939 2716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:58.598529 2716 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:58.598672 2716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:58.598836 2716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.6
I20240501 13:59:58.608258 2716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.17.6
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.6
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:58.609565 2716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:58.611378 2716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:58.623802 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:58.624462 2716 server_base.cc:1034] running on GCE node
W20240501 13:59:58.624895 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:58.625149 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:58.625834 2716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:58.626895 2716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:58.628103 2716 hybrid_clock.cc:648] HybridClock initialized: now 1714571998628052 us; error 45 us; skew 500 ppm
I20240501 13:59:58.628430 2716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:58.630717 2716 webserver.cc:460] Webserver started at http://127.28.17.6:37937/ using document root <none> and password file <none>
I20240501 13:59:58.631484 2716 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:58.631600 2716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:58.631943 2716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:58.636596 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 13:59:58.637378 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 13:59:58.642716 2716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 13:59:58.646189 2730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:58.647683 2716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:58.647850 2716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 13:59:58.648048 2716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:58.673918 2716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:58.674880 2716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:58.675154 2716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:58.676141 2716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:58.677685 2716 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:58.677780 2716 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:58.677894 2716 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:58.677963 2716 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:58.707466 2716 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.6:46037
I20240501 13:59:58.707542 2842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.6:46037 every 8 connection(s)
I20240501 13:59:58.708757 2716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 13:59:58.712292 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2716
I20240501 13:59:58.712456 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 13:59:58.715973 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.7:0
--local_ip_for_outbound_sockets=127.28.17.7
--webserver_interface=127.28.17.7
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:58.724397 2843 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:58.724716 2843 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:58.725651 2843 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:58.727275 2006 ts_manager.cc:194] Registered new tserver with Master: 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 13:59:58.728266 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.6:53127
I20240501 13:59:58.819623 2583 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
W20240501 13:59:59.059643 2846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:59.059942 2846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:59.060058 2846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:59.067618 2846 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:59.067745 2846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:59.067896 2846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.7
I20240501 13:59:59.076970 2846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.17.7
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.7
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:59.078311 2846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:59.080092 2846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:59.092468 2851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:59.093055 2852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:59.093722 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:59.094236 2846 server_base.cc:1034] running on GCE node
I20240501 13:59:59.094940 2846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:59.095950 2846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:59.097196 2846 hybrid_clock.cc:648] HybridClock initialized: now 1714571999097128 us; error 47 us; skew 500 ppm
I20240501 13:59:59.097543 2846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:59.099972 2846 webserver.cc:460] Webserver started at http://127.28.17.7:37329/ using document root <none> and password file <none>
I20240501 13:59:59.100744 2846 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:59.100860 2846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:59.101190 2846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:59.104049 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 13:59:59.104880 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 13:59:59.110435 2846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 13:59:59.114115 2860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:59.115644 2846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:59.115814 2846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 13:59:59.116014 2846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:59.132480 2846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:59.133379 2846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:59.133658 2846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:59.134658 2846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:59.136220 2846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:59.136312 2846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:59.136410 2846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:59.136472 2846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:59.166882 2846 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.7:41601
I20240501 13:59:59.166935 2972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.7:41601 every 8 connection(s)
I20240501 13:59:59.168215 2846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 13:59:59.178280 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2846
I20240501 13:59:59.178488 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 13:59:59.181037 2973 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:59.181351 2973 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:59.181830 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.8:0
--local_ip_for_outbound_sockets=127.28.17.8
--webserver_interface=127.28.17.8
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 13:59:59.182236 2973 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:59.183795 2006 ts_manager.cc:194] Registered new tserver with Master: c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 13:59:59.184592 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.7:33189
I20240501 13:59:59.267841 2713 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
W20240501 13:59:59.527817 2976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 13:59:59.528108 2976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 13:59:59.528200 2976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 13:59:59.535699 2976 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 13:59:59.535826 2976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 13:59:59.535938 2976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.8
I20240501 13:59:59.544983 2976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.17.8
--webserver_port=0
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.8
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 13:59:59.546241 2976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:59.547945 2976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:59.560257 2981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:59.560475 2982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:59.560878 2984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:59.561342 2976 server_base.cc:1034] running on GCE node
I20240501 13:59:59.561954 2976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 13:59:59.562986 2976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 13:59:59.564193 2976 hybrid_clock.cc:648] HybridClock initialized: now 1714571999564125 us; error 68 us; skew 500 ppm
I20240501 13:59:59.564515 2976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:59.566756 2976 webserver.cc:460] Webserver started at http://127.28.17.8:38337/ using document root <none> and password file <none>
I20240501 13:59:59.567516 2976 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:59.567627 2976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:59.567929 2976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:59.570540 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 13:59:59.571345 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 13:59:59.576718 2976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240501 13:59:59.580265 2990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:59.581691 2976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:59.581856 2976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 13:59:59.582054 2976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:59.609889 2976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:59.610903 2976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:59.611196 2976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:59.612267 2976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:59.613950 2976 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:59.614042 2976 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:59.614145 2976 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:59.614212 2976 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:59.643713 2976 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.8:34221
I20240501 13:59:59.643802 3102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.8:34221 every 8 connection(s)
I20240501 13:59:59.645001 2976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 13:59:59.646324 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 2976
I20240501 13:59:59.646521 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 13:59:59.658059 3103 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 13:59:59.658377 3103 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:59.659265 3103 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 13:59:59.660775 2006 ts_manager.cc:194] Registered new tserver with Master: 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 13:59:59.661856 2006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.8:34467
I20240501 13:59:59.663049 28740 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 13:59:59.680411 28740 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 13:59:59.680608 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2586
I20240501 13:59:59.689481 2708 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:59.731143 2843 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
I20240501 13:59:59.769506 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2586
I20240501 13:59:59.782009 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2716
I20240501 13:59:59.791126 2838 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:59.876147 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2716
I20240501 13:59:59.888358 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2846
I20240501 13:59:59.896589 2968 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 13:59:59.978044 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2846
I20240501 13:59:59.990271 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2976
I20240501 13:59:59.998484 3098 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:00:00.081372 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2976
I20240501 14:00:03.846859 2006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:03.891669 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.893519 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.893375 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 096a3e3778e9449ba5a12bbc91ef6486 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.894624 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096a3e3778e9449ba5a12bbc91ef6486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.895221 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.896744 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.899060 2517 tablet_service.cc:1467] Processing CreateTablet for tablet b3c4f120e9884911b9f6f5158f725bc2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.900154 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c4f120e9884911b9f6f5158f725bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.904057 2387 tablet_service.cc:1467] Processing CreateTablet for tablet b3c4f120e9884911b9f6f5158f725bc2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.904678 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.905256 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 096a3e3778e9449ba5a12bbc91ef6486 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.906173 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.906180 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096a3e3778e9449ba5a12bbc91ef6486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.910745 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c4f120e9884911b9f6f5158f725bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.912012 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 096a3e3778e9449ba5a12bbc91ef6486 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.912544 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096a3e3778e9449ba5a12bbc91ef6486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.919543 2125 tablet_service.cc:1467] Processing CreateTablet for tablet b3c4f120e9884911b9f6f5158f725bc2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:03.920043 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c4f120e9884911b9f6f5158f725bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:03.922216 3131 tablet_bootstrap.cc:492] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:03.926244 3132 tablet_bootstrap.cc:492] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:03.927223 3133 tablet_bootstrap.cc:492] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:03.930155 3132 tablet_bootstrap.cc:654] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.930770 3133 tablet_bootstrap.cc:654] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.931052 3134 tablet_bootstrap.cc:492] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:03.931401 3131 tablet_bootstrap.cc:654] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.932174 3133 log.cc:824] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:03.932641 3131 log.cc:824] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:03.932765 3132 log.cc:824] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:03.934868 3134 tablet_bootstrap.cc:654] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.936865 3134 log.cc:824] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:03.942070 3132 tablet_bootstrap.cc:492] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:03.942070 3134 tablet_bootstrap.cc:492] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:03.942107 3131 tablet_bootstrap.cc:492] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:03.942108 3133 tablet_bootstrap.cc:492] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:03.942427 3131 ts_tablet_manager.cc:1367] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.007s
I20240501 14:00:03.942427 3133 ts_tablet_manager.cc:1367] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240501 14:00:03.942422 3132 ts_tablet_manager.cc:1367] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20240501 14:00:03.942477 3134 ts_tablet_manager.cc:1367] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 14:00:03.951576 3132 raft_consensus.cc:348] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.951733 3131 raft_consensus.cc:348] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:03.951956 3134 raft_consensus.cc:348] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.952194 3131 raft_consensus.cc:374] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:03.952301 3131 raft_consensus.cc:729] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:03.952347 3134 raft_consensus.cc:374] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:03.952369 3132 raft_consensus.cc:374] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:03.952467 3132 raft_consensus.cc:729] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:03.952477 3134 raft_consensus.cc:729] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:03.952359 3133 raft_consensus.cc:348] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.952757 3133 raft_consensus.cc:374] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:03.952878 3133 raft_consensus.cc:729] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:03.953195 3132 consensus_queue.cc:260] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.953284 3134 consensus_queue.cc:260] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.953310 3131 consensus_queue.cc:260] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:03.953569 3133 consensus_queue.cc:260] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.962145 3131 ts_tablet_manager.cc:1398] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
I20240501 14:00:03.962850 3131 tablet_bootstrap.cc:492] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:03.965433 3142 raft_consensus.cc:482] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:03.965957 3131 tablet_bootstrap.cc:654] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.966276 3142 raft_consensus.cc:504] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.971333 3133 ts_tablet_manager.cc:1398] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.029s user 0.009s sys 0.008s
I20240501 14:00:03.971856 3133 tablet_bootstrap.cc:492] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:03.976379 3133 tablet_bootstrap.cc:654] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.976425 3134 ts_tablet_manager.cc:1398] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.034s user 0.013s sys 0.003s
I20240501 14:00:03.978539 3132 ts_tablet_manager.cc:1398] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.036s user 0.021s sys 0.013s
I20240501 14:00:03.979135 3134 tablet_bootstrap.cc:492] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:03.979121 3142 leader_election.cc:288] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:03.980930 3134 tablet_bootstrap.cc:654] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.982854 3132 tablet_bootstrap.cc:492] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:03.984611 3132 tablet_bootstrap.cc:654] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.988632 3131 tablet_bootstrap.cc:492] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:03.989780 3134 tablet_bootstrap.cc:492] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:03.990051 3134 ts_tablet_manager.cc:1367] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240501 14:00:03.991210 3134 raft_consensus.cc:348] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:03.991545 3134 raft_consensus.cc:374] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:03.991689 3134 raft_consensus.cc:729] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:03.991688 3131 ts_tablet_manager.cc:1367] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
I20240501 14:00:03.992002 3134 consensus_queue.cc:260] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:03.992627 3134 ts_tablet_manager.cc:1398] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:03.992951 3134 tablet_bootstrap.cc:492] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:03.994989 3134 tablet_bootstrap.cc:654] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:03.998833 3131 raft_consensus.cc:348] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:03.999131 3131 raft_consensus.cc:374] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:03.999222 3131 raft_consensus.cc:729] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:04.000864 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c4f120e9884911b9f6f5158f725bc2" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:04.001356 3141 raft_consensus.cc:482] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:04.001616 2147 raft_consensus.cc:2454] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:04.001608 3141 raft_consensus.cc:504] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:04.002544 2341 leader_election.cc:302] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:04.003051 3143 raft_consensus.cc:2781] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:04.003060 3131 consensus_queue.cc:260] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.003201 3143 raft_consensus.cc:482] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:04.003358 3143 raft_consensus.cc:3037] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.003962 3141 leader_election.cc:288] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:04.004566 3133 tablet_bootstrap.cc:492] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:04.004741 3133 ts_tablet_manager.cc:1367] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.008s
I20240501 14:00:04.005910 3133 raft_consensus.cc:348] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:04.006142 3133 raft_consensus.cc:374] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:04.006227 3133 raft_consensus.cc:729] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:04.006588 3143 raft_consensus.cc:504] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.007881 3143 leader_election.cc:288] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:04.008668 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c4f120e9884911b9f6f5158f725bc2" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:04.008936 2147 raft_consensus.cc:3037] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.006484 3133 consensus_queue.cc:260] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:04.010542 3131 ts_tablet_manager.cc:1398] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20240501 14:00:04.010643 3133 ts_tablet_manager.cc:1398] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240501 14:00:04.012009 2147 raft_consensus.cc:2454] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:04.012594 2341 leader_election.cc:302] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:04.013012 3143 raft_consensus.cc:2781] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:04.013165 3143 raft_consensus.cc:686] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:04.013700 3143 consensus_queue.cc:237] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.014019 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c4f120e9884911b9f6f5158f725bc2" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:04.014467 3132 tablet_bootstrap.cc:492] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:04.014654 3132 ts_tablet_manager.cc:1367] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.012s
I20240501 14:00:04.015168 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c4f120e9884911b9f6f5158f725bc2" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
W20240501 14:00:04.015283 2341 leader_election.cc:341] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:00:04.016362 2341 leader_election.cc:341] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:04.016544 3134 tablet_bootstrap.cc:492] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:04.016718 3134 ts_tablet_manager.cc:1367] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20240501 14:00:04.016873 3132 raft_consensus.cc:348] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.017102 3132 raft_consensus.cc:374] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:04.017199 3132 raft_consensus.cc:729] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:04.017418 3132 consensus_queue.cc:260] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.017433 3134 raft_consensus.cc:348] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.017664 3134 raft_consensus.cc:374] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:04.017745 3134 raft_consensus.cc:729] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:04.018009 3132 ts_tablet_manager.cc:1398] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:04.019012 3134 consensus_queue.cc:260] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.020107 2004 catalog_manager.cc:5484] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:04.021659 3134 ts_tablet_manager.cc:1398] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:04.026640 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096a3e3778e9449ba5a12bbc91ef6486" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:04.026906 2147 raft_consensus.cc:2454] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:04.027456 2211 leader_election.cc:302] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:04.027979 3151 raft_consensus.cc:2781] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:04.028157 3151 raft_consensus.cc:482] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:04.028299 3151 raft_consensus.cc:3037] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.031184 3151 raft_consensus.cc:504] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:04.031829 3151 leader_election.cc:288] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:04.031802 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096a3e3778e9449ba5a12bbc91ef6486" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:04.032299 2408 raft_consensus.cc:2454] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:04.032392 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096a3e3778e9449ba5a12bbc91ef6486" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:04.032450 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096a3e3778e9449ba5a12bbc91ef6486" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:04.032631 2147 raft_consensus.cc:3037] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.032744 2407 raft_consensus.cc:3037] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.034723 2147 raft_consensus.cc:2454] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:04.035029 2407 raft_consensus.cc:2454] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:04.035123 2211 leader_election.cc:302] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:04.035516 3151 raft_consensus.cc:2781] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:04.035715 3151 raft_consensus.cc:686] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:04.036196 3151 consensus_queue.cc:237] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:04.041317 2004 catalog_manager.cc:5484] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:04.054812 2584 tablet.cc:2369] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:04.091094 3139 raft_consensus.cc:482] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:04.091375 3139 raft_consensus.cc:504] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.093077 3139 leader_election.cc:288] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:00:04.094779 2454 tablet.cc:2369] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:04.100627 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f0041cd67f4f9baa2b4bf56b1cc2be" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:04.100742 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f0041cd67f4f9baa2b4bf56b1cc2be" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:04.100962 2147 raft_consensus.cc:2454] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:04.101015 2278 raft_consensus.cc:2454] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:04.101487 2471 leader_election.cc:302] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:04.101869 3139 raft_consensus.cc:2781] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:04.102020 3139 raft_consensus.cc:482] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:04.102133 3139 raft_consensus.cc:3037] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.104307 3139 raft_consensus.cc:504] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.105026 3139 leader_election.cc:288] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:04.105391 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f0041cd67f4f9baa2b4bf56b1cc2be" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:04.105594 2278 raft_consensus.cc:3037] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.105580 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f0041cd67f4f9baa2b4bf56b1cc2be" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:04.105770 2147 raft_consensus.cc:3037] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.107663 2147 raft_consensus.cc:2454] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:04.107667 2278 raft_consensus.cc:2454] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:04.108055 2471 leader_election.cc:302] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:04.108361 3139 raft_consensus.cc:2781] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:04.108759 3139 raft_consensus.cc:686] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:04.109228 3139 consensus_queue.cc:237] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.113878 2004 catalog_manager.cc:5484] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:04.138834 2324 tablet.cc:2369] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:00:04.187297 2193 tablet.cc:2369] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:04.300596 3139 raft_consensus.cc:482] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:04.300869 3139 raft_consensus.cc:504] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:04.301759 3139 leader_election.cc:288] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:04.303040 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c4f120e9884911b9f6f5158f725bc2" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:04.303386 2147 raft_consensus.cc:2379] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 745ac12076e84effa5bde1ecbd6a32ec in current term 1: Already voted for candidate a64f34785a7b44fa8070b0452a4e0ddd in this term.
I20240501 14:00:04.309358 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c4f120e9884911b9f6f5158f725bc2" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:04.310345 2471 leader_election.cc:302] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887
I20240501 14:00:04.310830 3139 raft_consensus.cc:3037] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:04.313766 3139 raft_consensus.cc:2726] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 14:00:04.472380 3160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:04.472769 3160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:04.480340 3160 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:00:04.577046 3139 consensus_queue.cc:1035] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:04.604568 3139 consensus_queue.cc:1035] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:04.629572 3184 consensus_queue.cc:1035] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:04.634531 3183 consensus_queue.cc:1035] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:04.642259 3189 consensus_queue.cc:1035] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:04.647608 3194 consensus_queue.cc:1035] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 334.313 ms
time per row: 0.167156 ms
I20240501 14:00:04.971782 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:04.980880 2257 tablet_service.cc:1467] Processing CreateTablet for tablet e5f7a0845388479b9d5f5f12d776c978 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.981109 2518 tablet_service.cc:1467] Processing CreateTablet for tablet e5f7a0845388479b9d5f5f12d776c978 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.981364 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f7a0845388479b9d5f5f12d776c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.981570 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f7a0845388479b9d5f5f12d776c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.982770 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.982851 2387 tablet_service.cc:1467] Processing CreateTablet for tablet e5f7a0845388479b9d5f5f12d776c978 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.983286 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 6c74bacbf4464bab9c403e2153953474 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.983448 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.983834 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.984544 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.987002 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 6c74bacbf4464bab9c403e2153953474 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.987425 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c74bacbf4464bab9c403e2153953474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.987511 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c74bacbf4464bab9c403e2153953474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.989969 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 6c74bacbf4464bab9c403e2153953474 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.990363 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c74bacbf4464bab9c403e2153953474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.991786 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f7a0845388479b9d5f5f12d776c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.995040 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:04.995559 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:04.997942 3241 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:05.000661 3238 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:05.001497 3241 tablet_bootstrap.cc:654] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.004197 3238 tablet_bootstrap.cc:654] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.006084 3239 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:05.006279 3240 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:05.007625 3241 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:05.007902 3241 ts_tablet_manager.cc:1367] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 14:00:05.008317 3240 tablet_bootstrap.cc:654] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.008365 3239 tablet_bootstrap.cc:654] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.008823 3241 raft_consensus.cc:348] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.009133 3241 raft_consensus.cc:374] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.009275 3241 raft_consensus.cc:729] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.009615 3241 consensus_queue.cc:260] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.011149 3241 ts_tablet_manager.cc:1398] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:05.011492 3241 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:05.016458 3241 tablet_bootstrap.cc:654] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.017048 3238 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:05.017102 3239 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:05.017215 3238 ts_tablet_manager.cc:1367] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240501 14:00:05.017387 3239 ts_tablet_manager.cc:1367] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 14:00:05.018237 3240 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:05.018095 3238 raft_consensus.cc:348] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.018316 3238 raft_consensus.cc:374] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.018234 3239 raft_consensus.cc:348] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.018414 3238 raft_consensus.cc:729] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.018532 3239 raft_consensus.cc:374] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.018668 3240 ts_tablet_manager.cc:1367] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240501 14:00:05.018800 3239 raft_consensus.cc:729] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.018714 3238 consensus_queue.cc:260] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.019345 3238 ts_tablet_manager.cc:1398] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:05.019505 3240 raft_consensus.cc:348] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.019706 3240 raft_consensus.cc:374] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.019793 3240 raft_consensus.cc:729] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.020072 3240 consensus_queue.cc:260] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.020083 3239 consensus_queue.cc:260] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.020745 3241 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:05.020864 3240 ts_tablet_manager.cc:1398] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:05.020932 3241 ts_tablet_manager.cc:1367] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20240501 14:00:05.021193 3240 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:05.021575 3239 ts_tablet_manager.cc:1398] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:05.021873 3239 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:05.022890 3240 tablet_bootstrap.cc:654] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.023406 3239 tablet_bootstrap.cc:654] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.023288 3241 raft_consensus.cc:348] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.023507 3241 raft_consensus.cc:374] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.023607 3241 raft_consensus.cc:729] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.023869 3241 consensus_queue.cc:260] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.024518 3241 ts_tablet_manager.cc:1398] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:05.024896 3241 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:05.026521 3241 tablet_bootstrap.cc:654] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.030107 3239 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:05.030298 3239 ts_tablet_manager.cc:1367] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240501 14:00:05.031108 3239 raft_consensus.cc:348] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.031342 3239 raft_consensus.cc:374] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.031409 3239 raft_consensus.cc:729] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.031651 3239 consensus_queue.cc:260] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.031857 3241 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:05.031983 3241 ts_tablet_manager.cc:1367] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 14:00:05.032186 3239 ts_tablet_manager.cc:1398] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:05.032255 3240 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:05.032425 3240 ts_tablet_manager.cc:1367] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 14:00:05.032550 3239 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:05.032712 3241 raft_consensus.cc:348] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.032899 3241 raft_consensus.cc:374] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.032994 3241 raft_consensus.cc:729] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.033219 3241 consensus_queue.cc:260] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.033736 3241 ts_tablet_manager.cc:1398] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:05.034338 3239 tablet_bootstrap.cc:654] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:05.034693 3240 raft_consensus.cc:348] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.035727 3240 raft_consensus.cc:374] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.035823 3240 raft_consensus.cc:729] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.036062 3240 consensus_queue.cc:260] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.036590 3239 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:05.036583 3240 ts_tablet_manager.cc:1398] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:05.036779 3239 ts_tablet_manager.cc:1367] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:05.037511 3239 raft_consensus.cc:348] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.037711 3239 raft_consensus.cc:374] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:05.037818 3239 raft_consensus.cc:729] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:05.038039 3239 consensus_queue.cc:260] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.038601 3239 ts_tablet_manager.cc:1398] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:05.045374 3184 raft_consensus.cc:482] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:05.045565 3184 raft_consensus.cc:504] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.046277 3184 leader_election.cc:288] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:05.047479 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:05.047731 2145 raft_consensus.cc:2454] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:05.048276 2341 leader_election.cc:302] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:05.048599 3184 raft_consensus.cc:2781] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:05.048691 3184 raft_consensus.cc:482] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:05.048769 3184 raft_consensus.cc:3037] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.051393 3184 raft_consensus.cc:504] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.052101 3184 leader_election.cc:288] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:05.052657 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:05.052845 2145 raft_consensus.cc:3037] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.053727 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:05.053877 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:05.054008 2278 raft_consensus.cc:2454] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:05.055212 2145 raft_consensus.cc:2454] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:05.055603 2341 leader_election.cc:302] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:05.055896 3184 raft_consensus.cc:2781] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:05.056047 3184 raft_consensus.cc:686] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:05.056286 3184 consensus_queue.cc:237] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.059015 2002 catalog_manager.cc:5484] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:05.076248 3184 raft_consensus.cc:482] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:05.076443 3184 raft_consensus.cc:504] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.077029 3184 leader_election.cc:288] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:05.077622 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c74bacbf4464bab9c403e2153953474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:05.077752 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c74bacbf4464bab9c403e2153953474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:05.077852 2277 raft_consensus.cc:2454] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:05.078014 2538 raft_consensus.cc:2454] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:05.078258 2341 leader_election.cc:302] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:05.078593 3184 raft_consensus.cc:2781] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:05.078712 3184 raft_consensus.cc:482] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:05.078802 3184 raft_consensus.cc:3037] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.080924 3184 raft_consensus.cc:504] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.081491 3184 leader_election.cc:288] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:05.081988 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c74bacbf4464bab9c403e2153953474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:05.082006 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c74bacbf4464bab9c403e2153953474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:05.082186 2538 raft_consensus.cc:3037] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.082181 2277 raft_consensus.cc:3037] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.084136 2277 raft_consensus.cc:2454] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:05.084143 2538 raft_consensus.cc:2454] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:05.084545 2341 leader_election.cc:302] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:05.084837 3184 raft_consensus.cc:2781] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:05.085000 3184 raft_consensus.cc:686] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:05.085268 3184 consensus_queue.cc:237] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:05.087978 2002 catalog_manager.cc:5484] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:05.294850 3184 raft_consensus.cc:482] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:05.295071 3184 raft_consensus.cc:504] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.295635 3184 leader_election.cc:288] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:05.296195 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f7a0845388479b9d5f5f12d776c978" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:05.296296 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f7a0845388479b9d5f5f12d776c978" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:05.296422 2277 raft_consensus.cc:2454] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:05.296519 2538 raft_consensus.cc:2454] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:05.296890 2341 leader_election.cc:302] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:05.297178 3184 raft_consensus.cc:2781] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:05.297271 3184 raft_consensus.cc:482] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:05.297372 3184 raft_consensus.cc:3037] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.299214 3184 raft_consensus.cc:504] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.299769 3184 leader_election.cc:288] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:05.300311 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f7a0845388479b9d5f5f12d776c978" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:05.300385 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f7a0845388479b9d5f5f12d776c978" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:05.300525 2277 raft_consensus.cc:3037] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.300606 2538 raft_consensus.cc:3037] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.303260 2277 raft_consensus.cc:2454] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:05.303259 2538 raft_consensus.cc:2454] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:05.303701 2341 leader_election.cc:302] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:05.303978 3184 raft_consensus.cc:2781] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:05.304175 3184 raft_consensus.cc:686] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:05.304432 3184 consensus_queue.cc:237] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:05.307062 2002 catalog_manager.cc:5484] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:05.544667 3184 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:05.550599 3188 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:05.619954 2277 raft_consensus.cc:3037] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:05.623594 3188 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:05.628297 3262 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 14:00:05.700625 3255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:05.700959 3255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:05.708400 3255 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:00:05.827970 3262 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:05.848129 3184 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.682 ms
time per row: 0.115341 ms
I20240501 14:00:06.088943 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:06.097857 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 147d90e667bb41b69be550df94446f33 (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.098011 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 147d90e667bb41b69be550df94446f33 (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.098371 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147d90e667bb41b69be550df94446f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.098449 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147d90e667bb41b69be550df94446f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.099067 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 809ef5a337774c8d98aea88a79e5152e (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.099421 2127 tablet_service.cc:1467] Processing CreateTablet for tablet c52127afcd764e03835b00dab69ced7c (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.099815 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52127afcd764e03835b00dab69ced7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.099750 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 147d90e667bb41b69be550df94446f33 (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.100152 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147d90e667bb41b69be550df94446f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.100126 2386 tablet_service.cc:1467] Processing CreateTablet for tablet c52127afcd764e03835b00dab69ced7c (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.100518 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52127afcd764e03835b00dab69ced7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.100943 2518 tablet_service.cc:1467] Processing CreateTablet for tablet c52127afcd764e03835b00dab69ced7c (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.101333 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52127afcd764e03835b00dab69ced7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.102880 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 809ef5a337774c8d98aea88a79e5152e (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.103258 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809ef5a337774c8d98aea88a79e5152e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.104606 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 809ef5a337774c8d98aea88a79e5152e (DEFAULT_TABLE table=rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:06.105085 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809ef5a337774c8d98aea88a79e5152e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.110076 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809ef5a337774c8d98aea88a79e5152e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:06.115564 3320 tablet_bootstrap.cc:492] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:06.117745 3320 tablet_bootstrap.cc:654] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.118705 3321 tablet_bootstrap.cc:492] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:06.120350 3321 tablet_bootstrap.cc:654] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.129097 3321 tablet_bootstrap.cc:492] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:06.129397 3321 ts_tablet_manager.cc:1367] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 14:00:06.130365 3321 raft_consensus.cc:348] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.130837 3321 raft_consensus.cc:374] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.130982 3321 raft_consensus.cc:729] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.131335 3321 consensus_queue.cc:260] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.132059 3321 ts_tablet_manager.cc:1398] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:06.132498 3321 tablet_bootstrap.cc:492] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:06.135288 3321 tablet_bootstrap.cc:654] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.136560 3324 tablet_bootstrap.cc:492] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:06.138326 3324 tablet_bootstrap.cc:654] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.143847 3321 tablet_bootstrap.cc:492] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:06.144104 3321 ts_tablet_manager.cc:1367] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240501 14:00:06.144222 3324 tablet_bootstrap.cc:492] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:06.144318 3319 tablet_bootstrap.cc:492] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:06.144481 3324 ts_tablet_manager.cc:1367] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:00:06.144922 3321 raft_consensus.cc:348] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.145112 3321 raft_consensus.cc:374] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.145217 3321 raft_consensus.cc:729] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.145364 3320 tablet_bootstrap.cc:492] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:06.145589 3320 ts_tablet_manager.cc:1367] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.014s
I20240501 14:00:06.145563 3321 consensus_queue.cc:260] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.146073 3319 tablet_bootstrap.cc:654] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.146332 3321 ts_tablet_manager.cc:1398] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:06.146414 3320 raft_consensus.cc:348] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.146687 3320 raft_consensus.cc:374] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.146735 3321 tablet_bootstrap.cc:492] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:06.146811 3320 raft_consensus.cc:729] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.145877 3324 raft_consensus.cc:348] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.147127 3324 raft_consensus.cc:374] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.147269 3324 raft_consensus.cc:729] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.147222 3320 consensus_queue.cc:260] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.147529 3324 consensus_queue.cc:260] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.148065 3324 ts_tablet_manager.cc:1398] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:06.148290 3321 tablet_bootstrap.cc:654] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.148546 3320 ts_tablet_manager.cc:1398] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:06.148977 3324 tablet_bootstrap.cc:492] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:06.150748 3324 tablet_bootstrap.cc:654] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.156632 3321 tablet_bootstrap.cc:492] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:06.156644 3319 tablet_bootstrap.cc:492] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:06.156698 3324 tablet_bootstrap.cc:492] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:06.156805 3319 ts_tablet_manager.cc:1367] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20240501 14:00:06.156805 3321 ts_tablet_manager.cc:1367] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240501 14:00:06.156857 3324 ts_tablet_manager.cc:1367] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:00:06.157497 3324 raft_consensus.cc:348] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.157689 3324 raft_consensus.cc:374] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.157582 3321 raft_consensus.cc:348] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.157582 3319 raft_consensus.cc:348] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.157764 3324 raft_consensus.cc:729] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.157783 3321 raft_consensus.cc:374] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.157783 3319 raft_consensus.cc:374] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.157867 3319 raft_consensus.cc:729] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.157866 3321 raft_consensus.cc:729] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.158084 3324 consensus_queue.cc:260] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.158085 3319 consensus_queue.cc:260] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.158094 3321 consensus_queue.cc:260] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.158658 3321 ts_tablet_manager.cc:1398] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:06.159371 3319 ts_tablet_manager.cc:1398] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:00:06.159628 3319 tablet_bootstrap.cc:492] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:06.159219 3324 ts_tablet_manager.cc:1398] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:06.160943 3319 tablet_bootstrap.cc:654] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.165364 3319 tablet_bootstrap.cc:492] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:06.165529 3319 ts_tablet_manager.cc:1367] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 14:00:06.166147 3319 raft_consensus.cc:348] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.166332 3319 raft_consensus.cc:374] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.166404 3319 raft_consensus.cc:729] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.166649 3319 consensus_queue.cc:260] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.167129 3319 ts_tablet_manager.cc:1398] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:06.167387 3319 tablet_bootstrap.cc:492] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:06.168843 3319 tablet_bootstrap.cc:654] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:06.170794 3319 tablet_bootstrap.cc:492] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:06.170943 3319 ts_tablet_manager.cc:1367] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:06.171605 3319 raft_consensus.cc:348] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.171756 3319 raft_consensus.cc:374] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:06.171836 3319 raft_consensus.cc:729] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:06.172036 3319 consensus_queue.cc:260] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.172564 3319 ts_tablet_manager.cc:1398] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:06.220800 3263 raft_consensus.cc:482] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:06.220989 3263 raft_consensus.cc:504] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.222437 3263 leader_election.cc:288] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:06.227049 3263 raft_consensus.cc:482] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:06.227267 3263 raft_consensus.cc:504] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.227986 3263 leader_election.cc:288] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:06.230710 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d90e667bb41b69be550df94446f33" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:06.230774 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52127afcd764e03835b00dab69ced7c" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:06.230823 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d90e667bb41b69be550df94446f33" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:06.231014 2277 raft_consensus.cc:2454] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:06.230998 2538 raft_consensus.cc:2454] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:06.231025 2537 raft_consensus.cc:2454] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:06.231559 2080 leader_election.cc:302] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:06.231966 3263 raft_consensus.cc:2781] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:06.231967 2080 leader_election.cc:302] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:06.232100 3263 raft_consensus.cc:482] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:06.232261 3263 raft_consensus.cc:3037] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.233985 3337 raft_consensus.cc:2781] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:06.234126 3337 raft_consensus.cc:482] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:06.234292 3337 raft_consensus.cc:3037] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.235126 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52127afcd764e03835b00dab69ced7c" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:06.235358 2407 raft_consensus.cc:2454] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:06.235425 3263 raft_consensus.cc:504] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.236055 3263 leader_election.cc:288] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:06.236493 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d90e667bb41b69be550df94446f33" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:06.236577 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147d90e667bb41b69be550df94446f33" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:06.236708 2537 raft_consensus.cc:3037] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.236769 2277 raft_consensus.cc:3037] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.237241 3337 raft_consensus.cc:504] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.237857 3337 leader_election.cc:288] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:06.238483 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52127afcd764e03835b00dab69ced7c" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:06.238539 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52127afcd764e03835b00dab69ced7c" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:06.238696 2407 raft_consensus.cc:3037] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.238732 2538 raft_consensus.cc:3037] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.239274 2277 raft_consensus.cc:2454] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:06.239290 2537 raft_consensus.cc:2454] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:06.239722 2080 leader_election.cc:302] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:06.240039 3337 raft_consensus.cc:2781] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:06.240213 3337 raft_consensus.cc:686] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:06.240677 3337 consensus_queue.cc:237] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:06.240926 2407 raft_consensus.cc:2454] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:06.241003 2538 raft_consensus.cc:2454] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:06.241357 2080 leader_election.cc:302] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:06.241607 3263 raft_consensus.cc:2781] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:06.242123 3263 raft_consensus.cc:686] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:06.242432 3263 consensus_queue.cc:237] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.244874 1997 catalog_manager.cc:5484] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:06.249368 1997 catalog_manager.cc:5484] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:06.514107 3195 raft_consensus.cc:482] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:06.514294 3195 raft_consensus.cc:504] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.515086 3195 leader_election.cc:288] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:06.515646 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ef5a337774c8d98aea88a79e5152e" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:06.515916 2145 raft_consensus.cc:2454] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:06.516376 2211 leader_election.cc:302] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:06.516714 3195 raft_consensus.cc:2781] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:06.516844 3195 raft_consensus.cc:482] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:06.516942 3195 raft_consensus.cc:3037] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.519052 3195 raft_consensus.cc:504] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.519755 3195 leader_election.cc:288] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:06.520185 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ef5a337774c8d98aea88a79e5152e" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:06.520427 2145 raft_consensus.cc:3037] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:06.521857 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ef5a337774c8d98aea88a79e5152e" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:06.521930 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809ef5a337774c8d98aea88a79e5152e" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:06.522121 2538 raft_consensus.cc:2454] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:06.522850 2145 raft_consensus.cc:2454] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:06.523227 2211 leader_election.cc:302] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters: 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:00:06.523519 3195 raft_consensus.cc:2781] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:06.523835 3195 raft_consensus.cc:686] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:06.524117 3195 consensus_queue.cc:237] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:06.526611 1997 catalog_manager.cc:5484] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:06.639308 3263 consensus_queue.cc:1035] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:06.646596 3263 consensus_queue.cc:1035] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:06.728807 3263 consensus_queue.cc:1035] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:06.733620 3337 consensus_queue.cc:1035] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:06.911141 3344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:06.911484 3344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:06.919328 3344 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:00:07.000181 2538 raft_consensus.cc:3037] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.003075 3195 consensus_queue.cc:1035] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:07.009541 3195 consensus_queue.cc:1035] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 220.5 ms
time per row: 0.11025 ms
I20240501 14:00:07.292502 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:07.301146 2125 tablet_service.cc:1467] Processing CreateTablet for tablet fd3f8661a13944098a18d3eea7ddeb00 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.301231 2256 tablet_service.cc:1467] Processing CreateTablet for tablet fd3f8661a13944098a18d3eea7ddeb00 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.301661 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f8661a13944098a18d3eea7ddeb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.301677 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f8661a13944098a18d3eea7ddeb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.302176 2387 tablet_service.cc:1467] Processing CreateTablet for tablet b6cca7b2b71045bf98e6eacd14e007d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.302651 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cca7b2b71045bf98e6eacd14e007d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.303277 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee13db8a347476fbdabdb402c93b584 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.303669 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee13db8a347476fbdabdb402c93b584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.307066 2517 tablet_service.cc:1467] Processing CreateTablet for tablet b6cca7b2b71045bf98e6eacd14e007d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.307137 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee13db8a347476fbdabdb402c93b584 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.307484 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cca7b2b71045bf98e6eacd14e007d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.307530 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee13db8a347476fbdabdb402c93b584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.302206 2386 tablet_service.cc:1467] Processing CreateTablet for tablet fd3f8661a13944098a18d3eea7ddeb00 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.311244 2126 tablet_service.cc:1467] Processing CreateTablet for tablet b6cca7b2b71045bf98e6eacd14e007d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.311520 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f8661a13944098a18d3eea7ddeb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.311764 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cca7b2b71045bf98e6eacd14e007d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.311683 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee13db8a347476fbdabdb402c93b584 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:07.312074 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee13db8a347476fbdabdb402c93b584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:07.323706 3405 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:07.326066 3405 tablet_bootstrap.cc:654] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.334560 3405 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:07.334841 3405 ts_tablet_manager.cc:1367] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 14:00:07.335467 3407 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:07.335907 3405 raft_consensus.cc:348] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.336244 3405 raft_consensus.cc:374] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.336387 3405 raft_consensus.cc:729] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.336382 3406 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:07.336727 3405 consensus_queue.cc:260] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.337407 3405 ts_tablet_manager.cc:1398] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:07.337702 3407 tablet_bootstrap.cc:654] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.338238 3404 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:07.338994 3405 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:07.340119 3404 tablet_bootstrap.cc:654] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.340469 3405 tablet_bootstrap.cc:654] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.346558 3407 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:07.346836 3407 ts_tablet_manager.cc:1367] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 14:00:07.347792 3407 raft_consensus.cc:348] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.348183 3407 raft_consensus.cc:374] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.348366 3407 raft_consensus.cc:729] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.348698 3407 consensus_queue.cc:260] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.349563 3407 ts_tablet_manager.cc:1398] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:07.349951 3407 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:07.351869 3407 tablet_bootstrap.cc:654] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.354115 3404 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:07.354329 3404 ts_tablet_manager.cc:1367] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20240501 14:00:07.355162 3404 raft_consensus.cc:348] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.355466 3404 raft_consensus.cc:374] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.355583 3404 raft_consensus.cc:729] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.355844 3404 consensus_queue.cc:260] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.356483 3404 ts_tablet_manager.cc:1398] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:07.356824 3404 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:07.358389 3404 tablet_bootstrap.cc:654] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.359385 3407 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:07.359553 3300 raft_consensus.cc:482] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:07.359634 3407 ts_tablet_manager.cc:1367] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:00:07.359717 3300 raft_consensus.cc:504] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.360051 3406 tablet_bootstrap.cc:654] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.360491 3300 leader_election.cc:288] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:07.360441 3407 raft_consensus.cc:348] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.360657 3407 raft_consensus.cc:374] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.360729 3407 raft_consensus.cc:729] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.360960 3407 consensus_queue.cc:260] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.361181 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:07.361407 2145 raft_consensus.cc:2454] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:07.361490 3407 ts_tablet_manager.cc:1398] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:07.361928 2341 leader_election.cc:302] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:07.362310 3407 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:07.362735 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:07.362960 2277 raft_consensus.cc:2454] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:07.363690 3300 raft_consensus.cc:2781] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:07.363806 3300 raft_consensus.cc:482] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:07.363896 3300 raft_consensus.cc:3037] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.365361 3407 tablet_bootstrap.cc:654] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.368692 3300 raft_consensus.cc:504] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.370025 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:07.370273 2145 raft_consensus.cc:3037] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.372841 3405 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:07.373034 3405 ts_tablet_manager.cc:1367] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.014s
I20240501 14:00:07.373971 3405 raft_consensus.cc:348] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.374222 3405 raft_consensus.cc:374] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.374300 3405 raft_consensus.cc:729] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.374588 3405 consensus_queue.cc:260] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.375178 3405 ts_tablet_manager.cc:1398] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:07.377025 2145 raft_consensus.cc:2454] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:07.377506 2341 leader_election.cc:302] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:07.378477 3300 leader_election.cc:288] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:07.378724 3300 raft_consensus.cc:2781] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:07.378846 3300 raft_consensus.cc:686] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:07.379153 3300 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:07.381425 3407 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:07.381583 3407 ts_tablet_manager.cc:1367] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.001s
I20240501 14:00:07.382411 3407 raft_consensus.cc:348] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.383955 3407 raft_consensus.cc:374] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.384050 3407 raft_consensus.cc:729] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.384265 3407 consensus_queue.cc:260] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.385807 3404 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:07.386011 3404 ts_tablet_manager.cc:1367] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.013s
I20240501 14:00:07.386902 3404 raft_consensus.cc:348] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.387153 3404 raft_consensus.cc:374] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.387228 3404 raft_consensus.cc:729] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.387502 3404 consensus_queue.cc:260] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.388092 3404 ts_tablet_manager.cc:1398] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:07.390542 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:07.390777 2277 raft_consensus.cc:3037] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.390867 3407 ts_tablet_manager.cc:1398] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240501 14:00:07.391037 3406 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:07.391198 3406 ts_tablet_manager.cc:1367] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.006s
I20240501 14:00:07.391810 3406 raft_consensus.cc:348] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.391953 3406 raft_consensus.cc:374] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.392006 3406 raft_consensus.cc:729] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.392204 3406 consensus_queue.cc:260] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.392743 3406 ts_tablet_manager.cc:1398] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:00:07.393085 3406 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:07.393486 2000 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:07.394595 2277 raft_consensus.cc:2454] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:07.394743 3406 tablet_bootstrap.cc:654] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:07.397791 3406 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:07.397944 3406 ts_tablet_manager.cc:1367] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:07.398756 3406 raft_consensus.cc:348] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.398977 3406 raft_consensus.cc:374] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:07.399078 3406 raft_consensus.cc:729] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:07.399324 3406 consensus_queue.cc:260] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.399818 3406 ts_tablet_manager.cc:1398] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:07.442425 3300 raft_consensus.cc:482] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:07.442687 3300 raft_consensus.cc:504] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.443312 3300 leader_election.cc:288] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:07.443853 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cca7b2b71045bf98e6eacd14e007d6" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:07.443904 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cca7b2b71045bf98e6eacd14e007d6" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:07.444087 2538 raft_consensus.cc:2454] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:07.444118 2145 raft_consensus.cc:2454] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:07.444530 2341 leader_election.cc:302] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:07.444830 3300 raft_consensus.cc:2781] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:07.444931 3300 raft_consensus.cc:482] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:07.445029 3300 raft_consensus.cc:3037] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.447153 3300 raft_consensus.cc:504] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.447744 3300 leader_election.cc:288] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:07.448276 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cca7b2b71045bf98e6eacd14e007d6" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:07.448282 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cca7b2b71045bf98e6eacd14e007d6" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:07.448505 2145 raft_consensus.cc:3037] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.448513 2538 raft_consensus.cc:3037] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.450806 2538 raft_consensus.cc:2454] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:07.451205 2341 leader_election.cc:302] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:07.451465 3300 raft_consensus.cc:2781] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:07.451576 2145 raft_consensus.cc:2454] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:07.451774 3300 raft_consensus.cc:686] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:07.452136 3300 consensus_queue.cc:237] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:07.454860 1997 catalog_manager.cc:5484] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:07.632213 3228 raft_consensus.cc:482] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:07.632392 3228 raft_consensus.cc:504] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.633080 3228 leader_election.cc:288] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:07.633693 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee13db8a347476fbdabdb402c93b584" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:07.633729 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee13db8a347476fbdabdb402c93b584" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:07.633942 2277 raft_consensus.cc:2454] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:07.633960 2407 raft_consensus.cc:2454] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:07.634348 2471 leader_election.cc:302] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:07.634639 3228 raft_consensus.cc:2781] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:07.634740 3228 raft_consensus.cc:482] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:07.634824 3228 raft_consensus.cc:3037] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.636670 3228 raft_consensus.cc:504] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.637184 3228 leader_election.cc:288] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:07.637634 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee13db8a347476fbdabdb402c93b584" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:07.637730 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee13db8a347476fbdabdb402c93b584" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:07.637898 2407 raft_consensus.cc:3037] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.637907 2277 raft_consensus.cc:3037] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:07.640632 2407 raft_consensus.cc:2454] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:07.640645 2277 raft_consensus.cc:2454] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:07.641114 2471 leader_election.cc:302] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:07.641415 3228 raft_consensus.cc:2781] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:07.641811 3228 raft_consensus.cc:686] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:07.642071 3228 consensus_queue.cc:237] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:07.644994 1997 catalog_manager.cc:5484] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:07.859567 3300 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:07.866025 3416 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:07.929519 3416 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:07.934720 3416 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:08.036164 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:08.036513 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:08.044098 3423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:00:08.132265 3228 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:08.143496 3421 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 226.48 ms
time per row: 0.11324 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:09.300271 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:09.309873 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 0b461bde7a84495f8828c69001a42944 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.309988 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 0b461bde7a84495f8828c69001a42944 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.310380 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b461bde7a84495f8828c69001a42944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.310398 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b461bde7a84495f8828c69001a42944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.311695 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 0b461bde7a84495f8828c69001a42944 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.312119 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b461bde7a84495f8828c69001a42944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.312184 2517 tablet_service.cc:1467] Processing CreateTablet for tablet bd0cfe423a214238ab93a6131545cef0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.312592 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0cfe423a214238ab93a6131545cef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.314929 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 504e8f4db6434b68a53dd52e08535090 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.315372 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 504e8f4db6434b68a53dd52e08535090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.318975 2127 tablet_service.cc:1467] Processing CreateTablet for tablet bd0cfe423a214238ab93a6131545cef0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.319432 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0cfe423a214238ab93a6131545cef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.322990 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 504e8f4db6434b68a53dd52e08535090 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.323486 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 504e8f4db6434b68a53dd52e08535090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.325511 3488 tablet_bootstrap.cc:492] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:09.327143 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 504e8f4db6434b68a53dd52e08535090 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.327477 3488 tablet_bootstrap.cc:654] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.327642 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 504e8f4db6434b68a53dd52e08535090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.329813 3489 tablet_bootstrap.cc:492] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:09.331779 2388 tablet_service.cc:1467] Processing CreateTablet for tablet bd0cfe423a214238ab93a6131545cef0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:09.332118 3489 tablet_bootstrap.cc:654] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.332299 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd0cfe423a214238ab93a6131545cef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:09.333949 3488 tablet_bootstrap.cc:492] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:09.334117 3488 ts_tablet_manager.cc:1367] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 14:00:09.334973 3488 raft_consensus.cc:348] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.335193 3488 raft_consensus.cc:374] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.335295 3488 raft_consensus.cc:729] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.335553 3488 consensus_queue.cc:260] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.335812 3376 raft_consensus.cc:482] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:09.335951 3376 raft_consensus.cc:504] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.336637 3376 leader_election.cc:288] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:09.339515 3488 ts_tablet_manager.cc:1398] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:09.342303 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b461bde7a84495f8828c69001a42944" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:09.345476 3488 tablet_bootstrap.cc:492] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:09.347321 3488 tablet_bootstrap.cc:654] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:09.347970 2211 leader_election.cc:341] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:00:09.353732 2211 leader_election.cc:341] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Not found: Tablet not found: 0b461bde7a84495f8828c69001a42944
I20240501 14:00:09.353925 2211 leader_election.cc:302] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887
I20240501 14:00:09.355245 3493 tablet_bootstrap.cc:492] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:09.357926 3487 tablet_bootstrap.cc:492] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:09.358644 3491 raft_consensus.cc:2726] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:09.358816 3493 tablet_bootstrap.cc:654] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.361757 3487 tablet_bootstrap.cc:654] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.366987 3493 tablet_bootstrap.cc:492] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:09.367234 3493 ts_tablet_manager.cc:1367] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240501 14:00:09.368024 3493 raft_consensus.cc:348] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.368326 3493 raft_consensus.cc:374] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.368440 3493 raft_consensus.cc:729] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.368736 3493 consensus_queue.cc:260] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.369534 3493 ts_tablet_manager.cc:1398] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:09.369872 3493 tablet_bootstrap.cc:492] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:09.371568 3493 tablet_bootstrap.cc:654] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.373898 3487 tablet_bootstrap.cc:492] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:09.374137 3487 ts_tablet_manager.cc:1367] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240501 14:00:09.375061 3487 raft_consensus.cc:348] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.375387 3487 raft_consensus.cc:374] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.375500 3487 raft_consensus.cc:729] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.375763 3487 consensus_queue.cc:260] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.376408 3487 ts_tablet_manager.cc:1398] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:09.376679 3487 tablet_bootstrap.cc:492] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:09.378610 3488 tablet_bootstrap.cc:492] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:09.378793 3488 ts_tablet_manager.cc:1367] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.006s
I20240501 14:00:09.379575 3488 raft_consensus.cc:348] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.379812 3488 raft_consensus.cc:374] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.379833 3487 tablet_bootstrap.cc:654] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.379894 3488 raft_consensus.cc:729] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.380132 3488 consensus_queue.cc:260] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.380657 3488 ts_tablet_manager.cc:1398] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:09.381206 3493 tablet_bootstrap.cc:492] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:09.381387 3493 ts_tablet_manager.cc:1367] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20240501 14:00:09.381754 3489 tablet_bootstrap.cc:492] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:09.381908 3489 ts_tablet_manager.cc:1367] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.015s
I20240501 14:00:09.382155 3493 raft_consensus.cc:348] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.382464 3493 raft_consensus.cc:374] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.382608 3493 raft_consensus.cc:729] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.382755 3489 raft_consensus.cc:348] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.382959 3489 raft_consensus.cc:374] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.382915 3493 consensus_queue.cc:260] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.383045 3489 raft_consensus.cc:729] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.383080 3487 tablet_bootstrap.cc:492] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:09.383229 3487 ts_tablet_manager.cc:1367] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:00:09.383281 3489 consensus_queue.cc:260] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.383599 3493 ts_tablet_manager.cc:1398] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:09.383777 3489 ts_tablet_manager.cc:1398] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:09.383914 3493 tablet_bootstrap.cc:492] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:09.384044 3489 tablet_bootstrap.cc:492] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:09.383944 3487 raft_consensus.cc:348] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.384148 3487 raft_consensus.cc:374] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.384212 3487 raft_consensus.cc:729] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.384485 3487 consensus_queue.cc:260] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.384997 3487 ts_tablet_manager.cc:1398] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:09.385630 3493 tablet_bootstrap.cc:654] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.385646 3489 tablet_bootstrap.cc:654] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:09.388151 3263 raft_consensus.cc:482] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:09.388321 3263 raft_consensus.cc:504] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.388926 3263 leader_election.cc:288] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:09.390072 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0cfe423a214238ab93a6131545cef0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:09.390311 2538 raft_consensus.cc:2454] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:09.390846 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0cfe423a214238ab93a6131545cef0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:09.391094 2407 raft_consensus.cc:2454] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:09.391571 2080 leader_election.cc:302] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:09.391896 3263 raft_consensus.cc:2781] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:09.391990 3263 raft_consensus.cc:482] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:09.392123 3263 raft_consensus.cc:3037] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.392928 3489 tablet_bootstrap.cc:492] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:09.393066 3493 tablet_bootstrap.cc:492] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:09.393091 3489 ts_tablet_manager.cc:1367] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20240501 14:00:09.393225 3493 ts_tablet_manager.cc:1367] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240501 14:00:09.393836 3489 raft_consensus.cc:348] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.394048 3489 raft_consensus.cc:374] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.394109 3489 raft_consensus.cc:729] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.394105 3493 raft_consensus.cc:348] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.394367 3493 raft_consensus.cc:374] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:09.394519 3493 raft_consensus.cc:729] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:09.394410 3489 consensus_queue.cc:260] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.394809 3493 consensus_queue.cc:260] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.394987 3489 ts_tablet_manager.cc:1398] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:09.395762 3493 ts_tablet_manager.cc:1398] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:09.394862 3263 raft_consensus.cc:504] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.396651 3263 leader_election.cc:288] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:09.398066 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0cfe423a214238ab93a6131545cef0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:09.398124 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd0cfe423a214238ab93a6131545cef0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:09.398299 2407 raft_consensus.cc:3037] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.398334 2537 raft_consensus.cc:3037] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.400264 2407 raft_consensus.cc:2454] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:09.400272 2537 raft_consensus.cc:2454] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:09.400626 2080 leader_election.cc:302] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:09.400892 3263 raft_consensus.cc:2781] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:09.401057 3263 raft_consensus.cc:686] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:09.401305 3263 consensus_queue.cc:237] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.403924 2001 catalog_manager.cc:5484] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:09.445351 3263 raft_consensus.cc:482] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:09.445536 3263 raft_consensus.cc:504] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.446204 3263 leader_election.cc:288] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:09.446671 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b461bde7a84495f8828c69001a42944" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:09.446823 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b461bde7a84495f8828c69001a42944" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:09.446924 2277 raft_consensus.cc:2454] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:09.447098 2407 raft_consensus.cc:2454] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:09.447412 2080 leader_election.cc:302] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:09.447659 3263 raft_consensus.cc:2781] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:09.447757 3263 raft_consensus.cc:482] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:09.447839 3263 raft_consensus.cc:3037] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.449744 3263 raft_consensus.cc:504] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.450273 3263 leader_election.cc:288] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:09.450865 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b461bde7a84495f8828c69001a42944" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:09.450915 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b461bde7a84495f8828c69001a42944" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:09.451092 2407 raft_consensus.cc:3037] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.451094 2277 raft_consensus.cc:3037] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.453233 2407 raft_consensus.cc:2454] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:09.453294 2277 raft_consensus.cc:2454] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:09.453629 2080 leader_election.cc:302] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:09.453897 3263 raft_consensus.cc:2781] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:09.454061 3263 raft_consensus.cc:686] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:09.454480 3263 consensus_queue.cc:237] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.457273 2001 catalog_manager.cc:5484] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:09.639875 3263 raft_consensus.cc:482] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:09.640100 3263 raft_consensus.cc:504] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.640771 3263 leader_election.cc:288] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:09.641324 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504e8f4db6434b68a53dd52e08535090" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:09.641389 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504e8f4db6434b68a53dd52e08535090" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:09.641579 2537 raft_consensus.cc:2454] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:09.641603 2277 raft_consensus.cc:2454] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:09.642019 2080 leader_election.cc:302] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:09.642459 3263 raft_consensus.cc:2781] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:09.642588 3263 raft_consensus.cc:482] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:09.642707 3263 raft_consensus.cc:3037] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.644722 3263 raft_consensus.cc:504] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.645349 3263 leader_election.cc:288] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:09.645722 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504e8f4db6434b68a53dd52e08535090" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:09.645859 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "504e8f4db6434b68a53dd52e08535090" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:09.645936 2537 raft_consensus.cc:3037] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.646059 2277 raft_consensus.cc:3037] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:09.648769 2537 raft_consensus.cc:2454] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:09.648768 2277 raft_consensus.cc:2454] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:09.649233 2080 leader_election.cc:302] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:09.649535 3263 raft_consensus.cc:2781] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:09.649719 3263 raft_consensus.cc:686] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:09.649978 3263 consensus_queue.cc:237] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:09.652652 2001 catalog_manager.cc:5484] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:09.906688 3501 consensus_queue.cc:1035] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:09.906797 3263 consensus_queue.cc:1035] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:09.935957 3510 consensus_queue.cc:1035] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:09.936640 3511 consensus_queue.cc:1035] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:10.082928 3510 consensus_queue.cc:1035] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:10.088579 3263 consensus_queue.cc:1035] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:10.091240 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:10.091668 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:10.100065 3505 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 203.747 ms
time per row: 0.101874 ms
I20240501 14:00:10.457577 1998 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:10.467041 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 73940622ad734b88af31e9fa0c1000e5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.467424 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 73940622ad734b88af31e9fa0c1000e5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.467567 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73940622ad734b88af31e9fa0c1000e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.467833 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73940622ad734b88af31e9fa0c1000e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.468906 2388 tablet_service.cc:1467] Processing CreateTablet for tablet b3738397b81b46868e09071be6f82334 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.469269 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3738397b81b46868e09071be6f82334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.470903 2256 tablet_service.cc:1467] Processing CreateTablet for tablet b3738397b81b46868e09071be6f82334 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.471285 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3738397b81b46868e09071be6f82334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.471208 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1f151470f24501857f38461d5b70c8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.471609 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1f151470f24501857f38461d5b70c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.472002 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 73940622ad734b88af31e9fa0c1000e5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.472368 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73940622ad734b88af31e9fa0c1000e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.474944 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1f151470f24501857f38461d5b70c8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.475351 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1f151470f24501857f38461d5b70c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.478978 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1f151470f24501857f38461d5b70c8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.479403 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1f151470f24501857f38461d5b70c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.480132 3566 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:10.480872 2517 tablet_service.cc:1467] Processing CreateTablet for tablet b3738397b81b46868e09071be6f82334 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:10.481288 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3738397b81b46868e09071be6f82334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:10.482095 3566 tablet_bootstrap.cc:654] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.490978 3568 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:10.493149 3568 tablet_bootstrap.cc:654] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.495421 3566 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:10.495666 3566 ts_tablet_manager.cc:1367] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20240501 14:00:10.497205 3565 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:10.499029 3565 tablet_bootstrap.cc:654] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.500926 3566 raft_consensus.cc:348] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.501379 3566 raft_consensus.cc:374] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.501510 3566 raft_consensus.cc:729] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.501854 3566 consensus_queue.cc:260] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.502010 3569 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:10.502790 3566 ts_tablet_manager.cc:1398] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 14:00:10.503074 3566 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:10.503387 3568 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:10.503535 3568 ts_tablet_manager.cc:1367] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240501 14:00:10.504029 3565 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:10.504166 3565 ts_tablet_manager.cc:1367] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:00:10.504678 3569 tablet_bootstrap.cc:654] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.504959 3565 raft_consensus.cc:348] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.505187 3565 raft_consensus.cc:374] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.505307 3565 raft_consensus.cc:729] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.505530 3568 raft_consensus.cc:348] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.505584 3565 consensus_queue.cc:260] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.505867 3568 raft_consensus.cc:374] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.506000 3568 raft_consensus.cc:729] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.506105 3565 ts_tablet_manager.cc:1398] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:10.506348 3568 consensus_queue.cc:260] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.508419 3568 ts_tablet_manager.cc:1398] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:10.508502 3566 tablet_bootstrap.cc:654] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.510839 3565 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:10.511373 3566 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:10.511554 3566 ts_tablet_manager.cc:1367] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 14:00:10.512312 3566 raft_consensus.cc:348] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.512540 3566 raft_consensus.cc:374] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.512621 3566 raft_consensus.cc:729] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.512779 3565 tablet_bootstrap.cc:654] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.513080 3569 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:10.512902 3566 consensus_queue.cc:260] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.513629 3569 ts_tablet_manager.cc:1367] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240501 14:00:10.514251 3566 ts_tablet_manager.cc:1398] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:10.514585 3566 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:10.514948 3569 raft_consensus.cc:348] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.515250 3569 raft_consensus.cc:374] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.515507 3569 raft_consensus.cc:729] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.515831 3569 consensus_queue.cc:260] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.515991 3565 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:10.516142 3566 tablet_bootstrap.cc:654] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.516171 3565 ts_tablet_manager.cc:1367] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:10.516523 3569 ts_tablet_manager.cc:1398] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:10.517341 3569 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:10.517577 3565 raft_consensus.cc:348] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.517839 3565 raft_consensus.cc:374] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.517940 3565 raft_consensus.cc:729] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.518208 3566 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:10.518357 3566 ts_tablet_manager.cc:1367] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:10.518378 3565 consensus_queue.cc:260] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.518968 3565 ts_tablet_manager.cc:1398] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:10.519163 3566 raft_consensus.cc:348] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.519354 3566 raft_consensus.cc:374] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.519459 3566 raft_consensus.cc:729] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.519630 3565 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:10.519703 3566 consensus_queue.cc:260] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.520238 3566 ts_tablet_manager.cc:1398] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:10.521245 3565 tablet_bootstrap.cc:654] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.522156 3569 tablet_bootstrap.cc:654] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:10.524982 3569 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:10.525290 3569 ts_tablet_manager.cc:1367] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240501 14:00:10.525583 3565 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:10.525730 3565 ts_tablet_manager.cc:1367] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:00:10.526160 3569 raft_consensus.cc:348] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.526371 3569 raft_consensus.cc:374] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.526441 3569 raft_consensus.cc:729] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.526368 3565 raft_consensus.cc:348] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.526610 3565 raft_consensus.cc:374] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:10.526697 3565 raft_consensus.cc:729] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:10.526757 3569 consensus_queue.cc:260] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.526906 3565 consensus_queue.cc:260] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.527299 3569 ts_tablet_manager.cc:1398] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:10.527364 3565 ts_tablet_manager.cc:1398] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:10.540067 3491 raft_consensus.cc:482] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:10.540284 3491 raft_consensus.cc:504] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.540906 3491 leader_election.cc:288] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:10.541473 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3738397b81b46868e09071be6f82334" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:10.541532 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3738397b81b46868e09071be6f82334" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:10.541694 2406 raft_consensus.cc:2454] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:10.541754 2537 raft_consensus.cc:2454] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:10.542085 2211 leader_election.cc:302] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:10.542358 3491 raft_consensus.cc:2781] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:10.542469 3491 raft_consensus.cc:482] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:10.542589 3491 raft_consensus.cc:3037] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.544387 3491 raft_consensus.cc:504] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.544930 3491 leader_election.cc:288] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:10.545395 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3738397b81b46868e09071be6f82334" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:10.545398 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3738397b81b46868e09071be6f82334" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:10.545578 2406 raft_consensus.cc:3037] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.545584 2537 raft_consensus.cc:3037] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.547443 2537 raft_consensus.cc:2454] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:10.547462 2406 raft_consensus.cc:2454] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:10.547869 2211 leader_election.cc:302] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:10.548110 3491 raft_consensus.cc:2781] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:10.548512 3491 raft_consensus.cc:686] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:10.548784 3491 consensus_queue.cc:237] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:10.551425 1997 catalog_manager.cc:5484] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:10.552960 3491 raft_consensus.cc:482] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:10.553094 3491 raft_consensus.cc:504] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.553720 3491 leader_election.cc:288] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:10.554196 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73940622ad734b88af31e9fa0c1000e5" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:10.554224 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73940622ad734b88af31e9fa0c1000e5" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:10.554435 2537 raft_consensus.cc:2454] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:10.554447 2406 raft_consensus.cc:2454] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:10.554822 2211 leader_election.cc:302] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:10.555040 3491 raft_consensus.cc:2781] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:10.555135 3491 raft_consensus.cc:482] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:10.555238 3491 raft_consensus.cc:3037] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.576330 3491 raft_consensus.cc:504] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.576979 3491 leader_election.cc:288] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:10.577668 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73940622ad734b88af31e9fa0c1000e5" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:10.577647 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73940622ad734b88af31e9fa0c1000e5" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:10.577884 2537 raft_consensus.cc:3037] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.577893 2406 raft_consensus.cc:3037] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.580646 2406 raft_consensus.cc:2454] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:10.580655 2537 raft_consensus.cc:2454] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:10.581148 2211 leader_election.cc:302] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:10.581490 3491 raft_consensus.cc:2781] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:10.581640 3491 raft_consensus.cc:686] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:10.581900 3491 consensus_queue.cc:237] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:10.584713 1998 catalog_manager.cc:5484] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:10.607635 3416 raft_consensus.cc:482] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:10.607877 3416 raft_consensus.cc:504] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.608645 3416 leader_election.cc:288] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:10.609099 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f151470f24501857f38461d5b70c8" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:10.609236 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f151470f24501857f38461d5b70c8" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:10.609311 2145 raft_consensus.cc:2454] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:10.609486 2277 raft_consensus.cc:2454] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:10.609750 2341 leader_election.cc:302] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:10.610014 3416 raft_consensus.cc:2781] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:10.610105 3416 raft_consensus.cc:482] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:10.610314 3416 raft_consensus.cc:3037] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.612362 3416 raft_consensus.cc:504] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.612897 3416 leader_election.cc:288] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:10.613375 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f151470f24501857f38461d5b70c8" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:10.613401 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f151470f24501857f38461d5b70c8" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:10.613574 2277 raft_consensus.cc:3037] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.613597 2145 raft_consensus.cc:3037] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:10.615451 2277 raft_consensus.cc:2454] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:10.615567 2145 raft_consensus.cc:2454] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:10.615861 2341 leader_election.cc:302] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:10.616133 3416 raft_consensus.cc:2781] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:10.616526 3416 raft_consensus.cc:686] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:10.616803 3416 consensus_queue.cc:237] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:10.619558 1998 catalog_manager.cc:5484] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:10.980964 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:10.981323 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:10.989130 3583 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:00:11.050104 3491 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:11.053179 3491 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:11.059983 3416 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:11.067035 3491 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:11.075278 3581 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:11.077287 3491 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 206.583 ms
time per row: 0.103292 ms
I20240501 14:00:11.394997 1998 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:11.403970 2126 tablet_service.cc:1467] Processing CreateTablet for tablet c60e9cd1e3a748aeb480df2ee8d97ba3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.404258 2258 tablet_service.cc:1467] Processing CreateTablet for tablet c60e9cd1e3a748aeb480df2ee8d97ba3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.404433 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60e9cd1e3a748aeb480df2ee8d97ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.404685 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60e9cd1e3a748aeb480df2ee8d97ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.405884 2517 tablet_service.cc:1467] Processing CreateTablet for tablet c60e9cd1e3a748aeb480df2ee8d97ba3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.405908 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.406345 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60e9cd1e3a748aeb480df2ee8d97ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.406613 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.406961 2256 tablet_service.cc:1467] Processing CreateTablet for tablet e16034fbf5ce43cebd5d1d3a217b79b3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.406984 2518 tablet_service.cc:1467] Processing CreateTablet for tablet e16034fbf5ce43cebd5d1d3a217b79b3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.407310 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16034fbf5ce43cebd5d1d3a217b79b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.407320 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16034fbf5ce43cebd5d1d3a217b79b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.411329 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.411793 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.414940 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.415354 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.418946 2125 tablet_service.cc:1467] Processing CreateTablet for tablet e16034fbf5ce43cebd5d1d3a217b79b3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:11.419420 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16034fbf5ce43cebd5d1d3a217b79b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:11.420073 3647 tablet_bootstrap.cc:492] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:11.421468 3646 tablet_bootstrap.cc:492] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:11.421808 3647 tablet_bootstrap.cc:654] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.421854 3645 tablet_bootstrap.cc:492] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:11.423259 3646 tablet_bootstrap.cc:654] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.423624 3645 tablet_bootstrap.cc:654] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.435145 3651 tablet_bootstrap.cc:492] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:11.437170 3651 tablet_bootstrap.cc:654] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.442116 3645 tablet_bootstrap.cc:492] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:11.442282 3645 ts_tablet_manager.cc:1367] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20240501 14:00:11.442529 3651 tablet_bootstrap.cc:492] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:11.442701 3651 ts_tablet_manager.cc:1367] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240501 14:00:11.443188 3645 raft_consensus.cc:348] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.443406 3645 raft_consensus.cc:374] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.443446 3647 tablet_bootstrap.cc:492] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:11.443593 3647 ts_tablet_manager.cc:1367] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20240501 14:00:11.443480 3651 raft_consensus.cc:348] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.443686 3651 raft_consensus.cc:374] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.443780 3651 raft_consensus.cc:729] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.444092 3651 consensus_queue.cc:260] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.444375 3647 raft_consensus.cc:348] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.444581 3647 raft_consensus.cc:374] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.444674 3647 raft_consensus.cc:729] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.444741 3651 ts_tablet_manager.cc:1398] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:11.444928 3647 consensus_queue.cc:260] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.445570 3647 ts_tablet_manager.cc:1398] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:11.445849 3647 tablet_bootstrap.cc:492] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:11.447501 3647 tablet_bootstrap.cc:654] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.450289 3646 tablet_bootstrap.cc:492] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:11.450549 3646 ts_tablet_manager.cc:1367] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.004s
I20240501 14:00:11.450899 3645 raft_consensus.cc:729] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.451750 3646 raft_consensus.cc:348] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.452111 3646 raft_consensus.cc:374] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.452275 3646 raft_consensus.cc:729] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.452550 3646 consensus_queue.cc:260] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.451416 3645 consensus_queue.cc:260] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.453363 3646 ts_tablet_manager.cc:1398] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:11.453866 3646 tablet_bootstrap.cc:492] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:11.454089 3645 ts_tablet_manager.cc:1398] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240501 14:00:11.454370 3645 tablet_bootstrap.cc:492] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:11.455742 3646 tablet_bootstrap.cc:654] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.456065 3645 tablet_bootstrap.cc:654] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.463084 3646 tablet_bootstrap.cc:492] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:11.463084 3645 tablet_bootstrap.cc:492] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:11.463274 3646 ts_tablet_manager.cc:1367] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240501 14:00:11.463274 3645 ts_tablet_manager.cc:1367] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 14:00:11.463299 3647 tablet_bootstrap.cc:492] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:11.463456 3647 ts_tablet_manager.cc:1367] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20240501 14:00:11.464040 3646 raft_consensus.cc:348] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.464051 3647 raft_consensus.cc:348] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.464033 3645 raft_consensus.cc:348] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.464251 3647 raft_consensus.cc:374] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.464254 3646 raft_consensus.cc:374] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.464257 3645 raft_consensus.cc:374] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.464324 3647 raft_consensus.cc:729] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.464324 3646 raft_consensus.cc:729] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.464325 3645 raft_consensus.cc:729] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.464581 3647 consensus_queue.cc:260] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.464581 3645 consensus_queue.cc:260] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.464577 3646 consensus_queue.cc:260] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.465030 3647 ts_tablet_manager.cc:1398] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 14:00:11.465101 3645 ts_tablet_manager.cc:1398] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:11.465297 3646 ts_tablet_manager.cc:1398] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:11.465337 3647 tablet_bootstrap.cc:492] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:11.465588 3646 tablet_bootstrap.cc:492] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:11.466941 3647 tablet_bootstrap.cc:654] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.466964 3646 tablet_bootstrap.cc:654] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:11.468786 3646 tablet_bootstrap.cc:492] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:11.468931 3646 ts_tablet_manager.cc:1367] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:11.469606 3646 raft_consensus.cc:348] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.469805 3646 raft_consensus.cc:374] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.470220 3646 raft_consensus.cc:729] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.470465 3646 consensus_queue.cc:260] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.471783 3646 ts_tablet_manager.cc:1398] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:11.475842 3647 tablet_bootstrap.cc:492] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:11.476012 3647 ts_tablet_manager.cc:1367] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240501 14:00:11.476675 3647 raft_consensus.cc:348] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.476833 3647 raft_consensus.cc:374] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:11.476895 3647 raft_consensus.cc:729] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:11.477087 3647 consensus_queue.cc:260] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.477555 3647 ts_tablet_manager.cc:1398] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:11.577277 3631 raft_consensus.cc:482] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:11.577584 3631 raft_consensus.cc:504] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.578361 3631 leader_election.cc:288] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:11.578943 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60e9cd1e3a748aeb480df2ee8d97ba3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:11.579041 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60e9cd1e3a748aeb480df2ee8d97ba3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:11.579222 2145 raft_consensus.cc:2454] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:11.579258 2536 raft_consensus.cc:2454] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:11.579689 2211 leader_election.cc:302] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:11.579967 3631 raft_consensus.cc:2781] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:11.580092 3631 raft_consensus.cc:482] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:11.580174 3631 raft_consensus.cc:3037] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.581954 3631 raft_consensus.cc:504] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.582552 3631 leader_election.cc:288] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:11.583022 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60e9cd1e3a748aeb480df2ee8d97ba3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:11.583106 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60e9cd1e3a748aeb480df2ee8d97ba3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:11.583248 2145 raft_consensus.cc:3037] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.583310 2536 raft_consensus.cc:3037] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.586020 2145 raft_consensus.cc:2454] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:11.586020 2536 raft_consensus.cc:2454] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:11.586481 2211 leader_election.cc:302] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:11.586824 3631 raft_consensus.cc:2781] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:11.586973 3631 raft_consensus.cc:686] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:11.587268 3631 consensus_queue.cc:237] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:11.590276 1998 catalog_manager.cc:5484] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:11.641564 3631 raft_consensus.cc:482] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:11.641777 3631 raft_consensus.cc:504] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.642349 3631 leader_election.cc:288] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:11.642915 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16034fbf5ce43cebd5d1d3a217b79b3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:11.643016 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16034fbf5ce43cebd5d1d3a217b79b3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:11.643143 2536 raft_consensus.cc:2454] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:11.643244 2145 raft_consensus.cc:2454] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:11.643541 2211 leader_election.cc:302] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:11.643833 3631 raft_consensus.cc:2781] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:11.643930 3631 raft_consensus.cc:482] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:11.644030 3631 raft_consensus.cc:3037] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.645793 3631 raft_consensus.cc:504] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.646345 3631 leader_election.cc:288] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:11.646796 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16034fbf5ce43cebd5d1d3a217b79b3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:11.646905 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16034fbf5ce43cebd5d1d3a217b79b3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:11.647022 2536 raft_consensus.cc:3037] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.647104 2145 raft_consensus.cc:3037] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.648900 2536 raft_consensus.cc:2454] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:11.648977 2145 raft_consensus.cc:2454] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:11.649276 2211 leader_election.cc:302] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:11.649554 3631 raft_consensus.cc:2781] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:11.649703 3631 raft_consensus.cc:686] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:11.649963 3631 consensus_queue.cc:237] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:11.652881 1998 catalog_manager.cc:5484] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:11.658736 3512 raft_consensus.cc:482] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:11.658921 3512 raft_consensus.cc:504] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.659588 3512 leader_election.cc:288] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:11.660091 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7498ef46b94171bcd84fc4ebfc3ae5" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:11.660200 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7498ef46b94171bcd84fc4ebfc3ae5" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:11.660326 2536 raft_consensus.cc:2454] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:11.660424 2406 raft_consensus.cc:2454] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:11.660751 2080 leader_election.cc:302] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:11.661021 3512 raft_consensus.cc:2781] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:11.661159 3512 raft_consensus.cc:482] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:11.661247 3512 raft_consensus.cc:3037] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.663168 3512 raft_consensus.cc:504] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.663718 3512 leader_election.cc:288] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:11.664218 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7498ef46b94171bcd84fc4ebfc3ae5" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:11.664286 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7498ef46b94171bcd84fc4ebfc3ae5" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:11.664407 2536 raft_consensus.cc:3037] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.664487 2406 raft_consensus.cc:3037] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:11.666266 2536 raft_consensus.cc:2454] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:11.666481 2406 raft_consensus.cc:2454] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:11.666757 2080 leader_election.cc:302] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:11.667101 3512 raft_consensus.cc:2781] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:11.667299 3512 raft_consensus.cc:686] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:11.667600 3512 consensus_queue.cc:237] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:11.671777 1998 catalog_manager.cc:5484] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:12.010306 3631 consensus_queue.cc:1035] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:12.018638 3578 consensus_queue.cc:1035] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:12.062561 3641 consensus_queue.cc:1035] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:12.063688 3641 consensus_queue.cc:1035] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:12.090047 3661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:12.090384 3661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:12.098387 3661 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:00:12.148603 3578 consensus_queue.cc:1035] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:12.154790 3666 consensus_queue.cc:1035] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 221.352 ms
time per row: 0.110676 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:13.335898 1998 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:13.345317 2127 tablet_service.cc:1467] Processing CreateTablet for tablet b9d8d993ce834c1db4dd4418f7a85780 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.345803 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8d993ce834c1db4dd4418f7a85780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.345897 2386 tablet_service.cc:1467] Processing CreateTablet for tablet b9d8d993ce834c1db4dd4418f7a85780 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.346293 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8d993ce834c1db4dd4418f7a85780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.346771 2258 tablet_service.cc:1467] Processing CreateTablet for tablet b9d8d993ce834c1db4dd4418f7a85780 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.347167 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8d993ce834c1db4dd4418f7a85780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.347312 2256 tablet_service.cc:1467] Processing CreateTablet for tablet d42a56dd51fe410cb42dd284ecbf2748 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.347680 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42a56dd51fe410cb42dd284ecbf2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.350901 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 87a79bfca7cd4ffcbb0e4448320d0576 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.351300 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a79bfca7cd4ffcbb0e4448320d0576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.351483 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 87a79bfca7cd4ffcbb0e4448320d0576 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.351867 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a79bfca7cd4ffcbb0e4448320d0576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.352108 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 87a79bfca7cd4ffcbb0e4448320d0576 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.352545 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a79bfca7cd4ffcbb0e4448320d0576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.355352 2388 tablet_service.cc:1467] Processing CreateTablet for tablet d42a56dd51fe410cb42dd284ecbf2748 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.355798 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42a56dd51fe410cb42dd284ecbf2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.357965 2517 tablet_service.cc:1467] Processing CreateTablet for tablet d42a56dd51fe410cb42dd284ecbf2748 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:13.358359 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42a56dd51fe410cb42dd284ecbf2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:13.368142 3730 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:13.368937 3732 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:13.370400 3730 tablet_bootstrap.cc:654] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.371037 3731 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:13.372092 3732 tablet_bootstrap.cc:654] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.375083 3731 tablet_bootstrap.cc:654] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.377125 3731 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:13.377354 3731 ts_tablet_manager.cc:1367] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:00:13.378250 3730 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:13.378528 3730 ts_tablet_manager.cc:1367] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:00:13.378613 3729 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:13.378566 3731 raft_consensus.cc:348] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.378808 3731 raft_consensus.cc:374] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.379317 3732 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:13.379495 3732 ts_tablet_manager.cc:1367] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240501 14:00:13.379527 3731 raft_consensus.cc:729] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.379448 3730 raft_consensus.cc:348] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.379640 3730 raft_consensus.cc:374] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.379765 3730 raft_consensus.cc:729] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.379833 3731 consensus_queue.cc:260] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.380072 3729 tablet_bootstrap.cc:654] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.380403 3732 raft_consensus.cc:348] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.380586 3731 ts_tablet_manager.cc:1398] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:13.380640 3732 raft_consensus.cc:374] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.380762 3732 raft_consensus.cc:729] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.380905 3731 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:13.381011 3732 consensus_queue.cc:260] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.381731 3732 ts_tablet_manager.cc:1398] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:13.382048 3732 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:13.382670 3731 tablet_bootstrap.cc:654] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.383878 3732 tablet_bootstrap.cc:654] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.388672 3732 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:13.388916 3732 ts_tablet_manager.cc:1367] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240501 14:00:13.391373 3732 raft_consensus.cc:348] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.391709 3732 raft_consensus.cc:374] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.391840 3732 raft_consensus.cc:729] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.392190 3732 consensus_queue.cc:260] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.392834 3732 ts_tablet_manager.cc:1398] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:00:13.380101 3730 consensus_queue.cc:260] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.393893 3730 ts_tablet_manager.cc:1398] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:00:13.394297 3730 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:13.394474 3731 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:13.394748 3731 ts_tablet_manager.cc:1367] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.001s
I20240501 14:00:13.395661 3731 raft_consensus.cc:348] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.395931 3731 raft_consensus.cc:374] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.396047 3731 raft_consensus.cc:729] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.396313 3731 consensus_queue.cc:260] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.396678 3730 tablet_bootstrap.cc:654] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.397037 3731 ts_tablet_manager.cc:1398] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:13.400929 3730 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:13.400946 3729 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:13.401089 3729 ts_tablet_manager.cc:1367] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.011s
I20240501 14:00:13.401155 3730 ts_tablet_manager.cc:1367] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240501 14:00:13.401883 3730 raft_consensus.cc:348] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.401849 3729 raft_consensus.cc:348] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.402066 3729 raft_consensus.cc:374] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.402168 3729 raft_consensus.cc:729] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.402276 3730 raft_consensus.cc:374] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.402412 3730 raft_consensus.cc:729] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.402436 3729 consensus_queue.cc:260] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.402766 3730 consensus_queue.cc:260] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.403375 3730 ts_tablet_manager.cc:1398] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:13.403667 3730 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:13.403717 3729 ts_tablet_manager.cc:1398] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:13.403954 3729 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:13.405313 3730 tablet_bootstrap.cc:654] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.406140 3729 tablet_bootstrap.cc:654] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:13.407598 3730 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:13.407761 3730 ts_tablet_manager.cc:1367] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:00:13.408347 3729 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:13.408496 3729 ts_tablet_manager.cc:1367] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:00:13.408452 3730 raft_consensus.cc:348] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.408639 3730 raft_consensus.cc:374] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.408725 3730 raft_consensus.cc:729] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.408938 3730 consensus_queue.cc:260] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.409169 3729 raft_consensus.cc:348] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.409395 3729 raft_consensus.cc:374] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:13.409437 3730 ts_tablet_manager.cc:1398] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:13.409480 3729 raft_consensus.cc:729] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:13.409713 3729 consensus_queue.cc:260] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.410315 3729 ts_tablet_manager.cc:1398] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:13.492048 3228 raft_consensus.cc:482] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:13.492240 3228 raft_consensus.cc:504] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.492870 3228 leader_election.cc:288] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:13.493399 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:13.493505 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:13.493626 2406 raft_consensus.cc:2454] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:13.493749 2277 raft_consensus.cc:2454] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:13.494115 2471 leader_election.cc:302] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:13.494422 3228 raft_consensus.cc:2781] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:13.494575 3228 raft_consensus.cc:482] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:13.494642 3228 raft_consensus.cc:3037] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.496628 3228 raft_consensus.cc:504] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.497176 3228 leader_election.cc:288] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:13.497639 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:13.497682 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:13.497829 2406 raft_consensus.cc:3037] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.497865 2277 raft_consensus.cc:3037] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.499750 2406 raft_consensus.cc:2454] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:13.499799 2277 raft_consensus.cc:2454] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:13.500197 2471 leader_election.cc:302] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:13.500516 3228 raft_consensus.cc:2781] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:13.500830 3228 raft_consensus.cc:686] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:13.501086 3228 consensus_queue.cc:237] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:13.503852 1998 catalog_manager.cc:5484] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:13.544587 3724 raft_consensus.cc:482] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:13.544787 3724 raft_consensus.cc:504] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.545398 3724 leader_election.cc:288] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:13.546007 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8d993ce834c1db4dd4418f7a85780" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:13.546084 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8d993ce834c1db4dd4418f7a85780" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:13.546231 2146 raft_consensus.cc:2454] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:13.546350 2277 raft_consensus.cc:2454] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:13.546692 2341 leader_election.cc:302] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:13.546948 3724 raft_consensus.cc:2781] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:13.547056 3724 raft_consensus.cc:482] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:13.547139 3724 raft_consensus.cc:3037] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.549106 3724 raft_consensus.cc:504] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.549682 3724 leader_election.cc:288] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:13.550235 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8d993ce834c1db4dd4418f7a85780" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:13.550280 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d8d993ce834c1db4dd4418f7a85780" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:13.550473 2146 raft_consensus.cc:3037] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.550485 2277 raft_consensus.cc:3037] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.553251 2146 raft_consensus.cc:2454] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:13.553249 2277 raft_consensus.cc:2454] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:13.553735 2341 leader_election.cc:302] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:13.554045 3724 raft_consensus.cc:2781] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:13.554437 3724 raft_consensus.cc:686] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:13.554764 3724 consensus_queue.cc:237] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:13.556170 3228 raft_consensus.cc:482] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:13.556363 3228 raft_consensus.cc:504] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.556946 3228 leader_election.cc:288] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:13.557360 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a79bfca7cd4ffcbb0e4448320d0576" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:13.557586 2406 raft_consensus.cc:2454] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:13.557540 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a79bfca7cd4ffcbb0e4448320d0576" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:13.557775 2146 raft_consensus.cc:2454] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:13.557983 2471 leader_election.cc:302] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:13.557770 1998 catalog_manager.cc:5484] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:13.558318 3228 raft_consensus.cc:2781] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:13.558427 3228 raft_consensus.cc:482] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:13.558559 3228 raft_consensus.cc:3037] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.560520 3228 raft_consensus.cc:504] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.561259 3228 leader_election.cc:288] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:13.561811 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a79bfca7cd4ffcbb0e4448320d0576" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:13.561887 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a79bfca7cd4ffcbb0e4448320d0576" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:13.562014 2406 raft_consensus.cc:3037] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.562086 2146 raft_consensus.cc:3037] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:13.563917 2406 raft_consensus.cc:2454] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:13.563983 2146 raft_consensus.cc:2454] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:13.564334 2471 leader_election.cc:302] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:13.564627 3228 raft_consensus.cc:2781] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:13.564793 3228 raft_consensus.cc:686] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:13.565057 3228 consensus_queue.cc:237] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:13.567979 1998 catalog_manager.cc:5484] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:13.952978 3747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:13.953311 3747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:13.960708 3747 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:00:14.001720 3724 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.008942 3772 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.015228 3228 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.029450 3742 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.073932 3742 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:14.088528 3780 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 212.983 ms
time per row: 0.106492 ms
I20240501 14:00:14.319391 2006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:14.328284 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 1124d6713d8344dd831e20f40b9d95fb (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.328481 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 1124d6713d8344dd831e20f40b9d95fb (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.328794 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1124d6713d8344dd831e20f40b9d95fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.328913 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1124d6713d8344dd831e20f40b9d95fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.329025 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed5ecc71ee94722adfd4d639a2cc270 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.329420 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed5ecc71ee94722adfd4d639a2cc270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.329720 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed5ecc71ee94722adfd4d639a2cc270 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.330093 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed5ecc71ee94722adfd4d639a2cc270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.331002 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 1124d6713d8344dd831e20f40b9d95fb (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.331414 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1124d6713d8344dd831e20f40b9d95fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.335736 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed5ecc71ee94722adfd4d639a2cc270 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.336150 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed5ecc71ee94722adfd4d639a2cc270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.340361 2388 tablet_service.cc:1467] Processing CreateTablet for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.340814 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.342937 2517 tablet_service.cc:1467] Processing CreateTablet for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.343343 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.347342 2125 tablet_service.cc:1467] Processing CreateTablet for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b (DEFAULT_TABLE table=rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:14.347741 3814 tablet_bootstrap.cc:492] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:14.348157 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:14.350097 3814 tablet_bootstrap.cc:654] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.358356 3816 tablet_bootstrap.cc:492] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:14.361227 3816 tablet_bootstrap.cc:654] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.368103 3814 tablet_bootstrap.cc:492] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:14.368304 3814 ts_tablet_manager.cc:1367] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20240501 14:00:14.369378 3818 tablet_bootstrap.cc:492] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:14.369196 3814 raft_consensus.cc:348] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.371081 3814 raft_consensus.cc:374] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.371198 3814 raft_consensus.cc:729] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.371883 3818 tablet_bootstrap.cc:654] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.371882 3814 consensus_queue.cc:260] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.374258 3814 ts_tablet_manager.cc:1398] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:14.374634 3814 tablet_bootstrap.cc:492] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:14.376190 3813 tablet_bootstrap.cc:492] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:14.378978 3814 tablet_bootstrap.cc:654] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.380498 3816 tablet_bootstrap.cc:492] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:14.380744 3816 ts_tablet_manager.cc:1367] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.017s
I20240501 14:00:14.381671 3816 raft_consensus.cc:348] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.382001 3816 raft_consensus.cc:374] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.382133 3816 raft_consensus.cc:729] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.382551 3816 consensus_queue.cc:260] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.383459 3816 ts_tablet_manager.cc:1398] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:14.383886 3816 tablet_bootstrap.cc:492] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:14.385751 3816 tablet_bootstrap.cc:654] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.386727 3813 tablet_bootstrap.cc:654] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.389282 3814 tablet_bootstrap.cc:492] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:14.389477 3814 ts_tablet_manager.cc:1367] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20240501 14:00:14.389971 3818 tablet_bootstrap.cc:492] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:14.390125 3818 ts_tablet_manager.cc:1367] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.004s
I20240501 14:00:14.390291 3814 raft_consensus.cc:348] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.390553 3814 raft_consensus.cc:374] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.390638 3814 raft_consensus.cc:729] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.390909 3818 raft_consensus.cc:348] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.390901 3814 consensus_queue.cc:260] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.391094 3818 raft_consensus.cc:374] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.391183 3818 raft_consensus.cc:729] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.391477 3814 ts_tablet_manager.cc:1398] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:14.391533 3818 consensus_queue.cc:260] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.391750 3792 raft_consensus.cc:482] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:14.391804 3814 tablet_bootstrap.cc:492] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:14.391866 3792 raft_consensus.cc:504] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.392591 3792 leader_election.cc:288] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:14.393371 3814 tablet_bootstrap.cc:654] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.393527 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed5ecc71ee94722adfd4d639a2cc270" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:14.393797 2145 raft_consensus.cc:2454] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:14.394209 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed5ecc71ee94722adfd4d639a2cc270" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:14.394872 2341 leader_election.cc:302] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
W20240501 14:00:14.395632 2341 leader_election.cc:341] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:14.395967 3792 raft_consensus.cc:2781] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:14.396071 3792 raft_consensus.cc:482] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:14.396147 3792 raft_consensus.cc:3037] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.396471 3816 tablet_bootstrap.cc:492] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:14.396639 3816 ts_tablet_manager.cc:1367] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20240501 14:00:14.397439 3816 raft_consensus.cc:348] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.397673 3816 raft_consensus.cc:374] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.397749 3816 raft_consensus.cc:729] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.397994 3816 consensus_queue.cc:260] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.398473 3816 ts_tablet_manager.cc:1398] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:14.399420 3792 raft_consensus.cc:504] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.399844 3813 tablet_bootstrap.cc:492] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:14.399996 3813 ts_tablet_manager.cc:1367] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.005s
I20240501 14:00:14.400578 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed5ecc71ee94722adfd4d639a2cc270" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:14.400808 2145 raft_consensus.cc:3037] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.400774 3813 raft_consensus.cc:348] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.400964 3813 raft_consensus.cc:374] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.401053 3813 raft_consensus.cc:729] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.401268 3813 consensus_queue.cc:260] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.401932 3813 ts_tablet_manager.cc:1398] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:14.402220 3813 tablet_bootstrap.cc:492] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:14.403713 3814 tablet_bootstrap.cc:492] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:14.403867 3814 ts_tablet_manager.cc:1367] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 14:00:14.403873 3813 tablet_bootstrap.cc:654] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.404484 2145 raft_consensus.cc:2454] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:14.404651 3814 raft_consensus.cc:348] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.404850 3814 raft_consensus.cc:374] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.404950 3814 raft_consensus.cc:729] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.405088 2341 leader_election.cc:302] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:14.405387 3772 raft_consensus.cc:2781] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:14.405958 3772 raft_consensus.cc:686] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:14.406320 3813 tablet_bootstrap.cc:492] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:14.405218 3814 consensus_queue.cc:260] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.406282 3772 consensus_queue.cc:237] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.406477 3813 ts_tablet_manager.cc:1367] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:00:14.406541 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed5ecc71ee94722adfd4d639a2cc270" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:14.406829 3814 ts_tablet_manager.cc:1398] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:14.407059 3792 leader_election.cc:288] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:14.407263 3813 raft_consensus.cc:348] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.407469 3813 raft_consensus.cc:374] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 14:00:14.407490 2341 leader_election.cc:341] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:14.407549 3813 raft_consensus.cc:729] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.407795 3813 consensus_queue.cc:260] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.408396 3813 ts_tablet_manager.cc:1398] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:14.410359 3818 ts_tablet_manager.cc:1398] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:00:14.410679 3818 tablet_bootstrap.cc:492] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:14.411337 2000 catalog_manager.cc:5484] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:14.412287 3818 tablet_bootstrap.cc:654] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:14.413923 3818 tablet_bootstrap.cc:492] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:14.414103 3818 ts_tablet_manager.cc:1367] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:14.414772 3818 raft_consensus.cc:348] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.414948 3818 raft_consensus.cc:374] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:14.415027 3818 raft_consensus.cc:729] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:14.415272 3818 consensus_queue.cc:260] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.415776 3818 ts_tablet_manager.cc:1398] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:14.568876 3748 raft_consensus.cc:482] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:14.569080 3748 raft_consensus.cc:504] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.569849 3748 leader_election.cc:288] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:14.570395 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1124d6713d8344dd831e20f40b9d95fb" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:14.570405 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1124d6713d8344dd831e20f40b9d95fb" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:14.570660 2277 raft_consensus.cc:2454] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:14.570680 2537 raft_consensus.cc:2454] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:14.571112 2080 leader_election.cc:302] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:14.571427 3748 raft_consensus.cc:2781] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:14.571532 3748 raft_consensus.cc:482] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:14.571614 3748 raft_consensus.cc:3037] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.573499 3748 raft_consensus.cc:504] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.574030 3748 leader_election.cc:288] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:14.574671 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1124d6713d8344dd831e20f40b9d95fb" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:14.574888 2537 raft_consensus.cc:3037] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.574792 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1124d6713d8344dd831e20f40b9d95fb" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:14.575026 2277 raft_consensus.cc:3037] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.577193 2537 raft_consensus.cc:2454] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:14.577307 2277 raft_consensus.cc:2454] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:14.577615 2080 leader_election.cc:302] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:14.577893 3748 raft_consensus.cc:2781] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:14.578305 3748 raft_consensus.cc:686] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:14.578660 3748 consensus_queue.cc:237] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:14.581923 2000 catalog_manager.cc:5484] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:14.684563 3792 raft_consensus.cc:482] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:14.684751 3792 raft_consensus.cc:504] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.685374 3792 leader_election.cc:288] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:14.685846 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d64c0e7fb4852b4cdf9bec0eb6a6b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:14.685932 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d64c0e7fb4852b4cdf9bec0eb6a6b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:14.686057 2145 raft_consensus.cc:2454] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:14.686151 2537 raft_consensus.cc:2454] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:14.686564 2341 leader_election.cc:302] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:14.686836 3792 raft_consensus.cc:2781] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:14.686949 3792 raft_consensus.cc:482] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:14.687038 3792 raft_consensus.cc:3037] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.688928 3792 raft_consensus.cc:504] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.689450 3792 leader_election.cc:288] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:14.689947 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d64c0e7fb4852b4cdf9bec0eb6a6b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:14.689962 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d64c0e7fb4852b4cdf9bec0eb6a6b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:14.690133 2145 raft_consensus.cc:3037] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.690131 2537 raft_consensus.cc:3037] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.691949 2537 raft_consensus.cc:2454] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:14.692036 2145 raft_consensus.cc:2454] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:14.692391 2341 leader_election.cc:302] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:14.692694 3792 raft_consensus.cc:2781] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:14.692828 3792 raft_consensus.cc:686] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:14.693101 3792 consensus_queue.cc:237] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:14.696034 2000 catalog_manager.cc:5484] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:14.762053 3578 raft_consensus.cc:482] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:14.762284 3578 raft_consensus.cc:504] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:14.763113 3578 leader_election.cc:288] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:14.763722 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed5ecc71ee94722adfd4d639a2cc270" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:14.763733 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed5ecc71ee94722adfd4d639a2cc270" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:14.763984 2145 raft_consensus.cc:2379] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in current term 1: Already voted for candidate a64f34785a7b44fa8070b0452a4e0ddd in this term.
I20240501 14:00:14.764443 2211 leader_election.cc:302] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887
I20240501 14:00:14.764709 3578 raft_consensus.cc:3037] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:14.766731 3578 raft_consensus.cc:2726] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:15.010412 3772 consensus_queue.cc:1035] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:15.016790 3792 consensus_queue.cc:1035] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:15.074143 3831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:15.074532 3831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:15.082017 3831 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:15.092778 3827 consensus_queue.cc:1035] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:15.100991 3848 consensus_queue.cc:1035] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240501 14:00:15.252816 3772 consensus_queue.cc:1035] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:15.273121 3835 consensus_queue.cc:1035] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:15.283993 3871 mvcc.cc:205] Tried to move back new op lower bound from 7022886974530150400 to 7022886972185219072. Current Snapshot: MvccSnapshot[applied={T|T < 7022886974530150400}]
I20240501 14:00:15.285734 3852 mvcc.cc:205] Tried to move back new op lower bound from 7022886974530150400 to 7022886972185219072. Current Snapshot: MvccSnapshot[applied={T|T < 7022886974530150400}]
INSERT report
rows total: 2000
time total: 210.315 ms
time per row: 0.105158 ms
I20240501 14:00:15.473011 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:15.481810 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 304b465c0a99456790bfc8f262b61ecd (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.482208 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 304b465c0a99456790bfc8f262b61ecd (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.482303 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304b465c0a99456790bfc8f262b61ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.482600 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304b465c0a99456790bfc8f262b61ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.487565 2516 tablet_service.cc:1467] Processing CreateTablet for tablet f1e048df08cf4f7eaacb78d6592d79b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.487744 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 304b465c0a99456790bfc8f262b61ecd (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.487905 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 1298bb5c8df3430b95d01ad23ff9cb71 (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.488042 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e048df08cf4f7eaacb78d6592d79b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.488159 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304b465c0a99456790bfc8f262b61ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.488289 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1298bb5c8df3430b95d01ad23ff9cb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.491067 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 1298bb5c8df3430b95d01ad23ff9cb71 (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.491500 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1298bb5c8df3430b95d01ad23ff9cb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.494275 2388 tablet_service.cc:1467] Processing CreateTablet for tablet f1e048df08cf4f7eaacb78d6592d79b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.494711 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e048df08cf4f7eaacb78d6592d79b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.495152 3895 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:15.496862 3895 tablet_bootstrap.cc:654] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.498595 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 1298bb5c8df3430b95d01ad23ff9cb71 (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.498800 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f1e048df08cf4f7eaacb78d6592d79b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:15.499078 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1298bb5c8df3430b95d01ad23ff9cb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.499276 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e048df08cf4f7eaacb78d6592d79b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:15.502233 3895 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:15.502403 3895 ts_tablet_manager.cc:1367] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:15.503258 3895 raft_consensus.cc:348] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.503494 3895 raft_consensus.cc:374] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.503587 3895 raft_consensus.cc:729] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.507539 3896 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:15.507555 3895 consensus_queue.cc:260] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.507726 3894 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:15.509290 3895 ts_tablet_manager.cc:1398] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:15.510529 3894 tablet_bootstrap.cc:654] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.511108 3896 tablet_bootstrap.cc:654] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.517379 3898 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:15.519672 3898 tablet_bootstrap.cc:654] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.520283 3896 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:15.520403 3894 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:15.520450 3896 ts_tablet_manager.cc:1367] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240501 14:00:15.520560 3894 ts_tablet_manager.cc:1367] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240501 14:00:15.521140 3894 raft_consensus.cc:348] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.521309 3894 raft_consensus.cc:374] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.521410 3894 raft_consensus.cc:729] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.521312 3896 raft_consensus.cc:348] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.521517 3896 raft_consensus.cc:374] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.521651 3896 raft_consensus.cc:729] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.521657 3894 consensus_queue.cc:260] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.521934 3896 consensus_queue.cc:260] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.522234 3894 ts_tablet_manager.cc:1398] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:15.522686 3894 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:15.523248 3896 ts_tablet_manager.cc:1398] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:15.523577 3896 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:15.524473 3894 tablet_bootstrap.cc:654] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.525288 3896 tablet_bootstrap.cc:654] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.526283 3742 raft_consensus.cc:482] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:15.526439 3742 raft_consensus.cc:504] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.527248 3742 leader_election.cc:288] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:15.528733 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298bb5c8df3430b95d01ad23ff9cb71" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:15.529018 2145 raft_consensus.cc:2454] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:15.529786 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298bb5c8df3430b95d01ad23ff9cb71" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
W20240501 14:00:15.530411 2471 leader_election.cc:341] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:15.531718 3896 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:15.531980 3898 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:15.532145 3898 ts_tablet_manager.cc:1367] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240501 14:00:15.531965 3896 ts_tablet_manager.cc:1367] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 14:00:15.532967 3898 raft_consensus.cc:348] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.533193 3898 raft_consensus.cc:374] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.533299 3898 raft_consensus.cc:729] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.533453 3896 raft_consensus.cc:348] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.533732 3896 raft_consensus.cc:374] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.533545 3898 consensus_queue.cc:260] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.533885 3896 raft_consensus.cc:729] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.534195 3896 consensus_queue.cc:260] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.532150 2471 leader_election.cc:302] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:15.535092 3898 ts_tablet_manager.cc:1398] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:15.535423 3898 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:15.536234 3742 raft_consensus.cc:2781] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:15.536360 3742 raft_consensus.cc:482] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:15.536449 3742 raft_consensus.cc:3037] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.536643 3894 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:15.536814 3894 ts_tablet_manager.cc:1367] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20240501 14:00:15.536979 3898 tablet_bootstrap.cc:654] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.537575 3894 raft_consensus.cc:348] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.537801 3894 raft_consensus.cc:374] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.537889 3894 raft_consensus.cc:729] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.538144 3894 consensus_queue.cc:260] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.538872 3894 ts_tablet_manager.cc:1398] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:00:15.539227 3894 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:15.544102 3894 tablet_bootstrap.cc:654] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.544672 3742 raft_consensus.cc:504] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.545845 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298bb5c8df3430b95d01ad23ff9cb71" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:15.546087 2145 raft_consensus.cc:3037] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.549296 2145 raft_consensus.cc:2454] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:15.549870 2471 leader_election.cc:302] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:15.551345 3906 raft_consensus.cc:2781] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:15.551595 3906 raft_consensus.cc:686] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:15.551980 3906 consensus_queue.cc:237] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:15.553453 3742 leader_election.cc:288] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:15.563817 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298bb5c8df3430b95d01ad23ff9cb71" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:15.564091 2278 raft_consensus.cc:3037] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.565366 2004 catalog_manager.cc:5484] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:15.570340 2278 raft_consensus.cc:2454] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:15.574144 3896 ts_tablet_manager.cc:1398] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.042s user 0.010s sys 0.020s
I20240501 14:00:15.574568 3896 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:15.575024 3898 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:15.575295 3898 ts_tablet_manager.cc:1367] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.034s
I20240501 14:00:15.576239 3896 tablet_bootstrap.cc:654] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:15.576124 3898 raft_consensus.cc:348] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.576370 3898 raft_consensus.cc:374] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.576453 3898 raft_consensus.cc:729] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.576714 3898 consensus_queue.cc:260] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.577214 3894 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:15.577368 3894 ts_tablet_manager.cc:1367] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.005s
I20240501 14:00:15.577935 3898 ts_tablet_manager.cc:1398] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:15.578130 3894 raft_consensus.cc:348] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.578323 3894 raft_consensus.cc:374] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.578372 3896 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:15.578387 3894 raft_consensus.cc:729] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.578541 3896 ts_tablet_manager.cc:1367] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:00:15.578612 3894 consensus_queue.cc:260] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.579109 3894 ts_tablet_manager.cc:1398] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:15.579273 3896 raft_consensus.cc:348] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.579471 3896 raft_consensus.cc:374] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:15.579557 3896 raft_consensus.cc:729] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:15.579882 3896 consensus_queue.cc:260] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.580406 3896 ts_tablet_manager.cc:1398] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:15.693696 3904 raft_consensus.cc:482] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:15.693951 3904 raft_consensus.cc:504] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.694751 3904 leader_election.cc:288] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:15.695361 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b465c0a99456790bfc8f262b61ecd" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:15.695520 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b465c0a99456790bfc8f262b61ecd" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:15.695623 2278 raft_consensus.cc:2454] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:15.695802 2408 raft_consensus.cc:2454] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:15.696102 2471 leader_election.cc:302] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:15.696535 3904 raft_consensus.cc:2781] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:15.696676 3904 raft_consensus.cc:482] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:15.696779 3904 raft_consensus.cc:3037] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.698853 3904 raft_consensus.cc:504] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.699468 3904 leader_election.cc:288] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:15.700137 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b465c0a99456790bfc8f262b61ecd" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:15.700351 2278 raft_consensus.cc:3037] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.700299 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b465c0a99456790bfc8f262b61ecd" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:15.700515 2408 raft_consensus.cc:3037] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.702412 2278 raft_consensus.cc:2454] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:15.702769 2408 raft_consensus.cc:2454] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:15.702900 2471 leader_election.cc:302] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:15.703200 3904 raft_consensus.cc:2781] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:15.703362 3904 raft_consensus.cc:686] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:15.703616 3904 consensus_queue.cc:237] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.706954 2003 catalog_manager.cc:5484] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:15.786994 3792 raft_consensus.cc:482] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:15.787191 3792 raft_consensus.cc:504] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.787878 3792 leader_election.cc:288] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:15.788369 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e048df08cf4f7eaacb78d6592d79b1" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:15.788411 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e048df08cf4f7eaacb78d6592d79b1" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:15.788568 2278 raft_consensus.cc:2454] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:15.788621 2537 raft_consensus.cc:2454] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:15.788925 2341 leader_election.cc:302] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:15.789197 3792 raft_consensus.cc:2781] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:15.789314 3792 raft_consensus.cc:482] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:15.789389 3792 raft_consensus.cc:3037] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.791230 3792 raft_consensus.cc:504] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.791760 3792 leader_election.cc:288] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:15.792217 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e048df08cf4f7eaacb78d6592d79b1" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:15.792264 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e048df08cf4f7eaacb78d6592d79b1" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:15.792428 2537 raft_consensus.cc:3037] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.792464 2278 raft_consensus.cc:3037] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:15.794344 2278 raft_consensus.cc:2454] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:15.794492 2537 raft_consensus.cc:2454] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:15.794811 2341 leader_election.cc:302] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:15.795090 3792 raft_consensus.cc:2781] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:15.795301 3792 raft_consensus.cc:686] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:15.795606 3792 consensus_queue.cc:237] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:15.798781 2003 catalog_manager.cc:5484] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:16.063248 3904 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:16.070832 3906 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:16.168236 3913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:16.168574 3913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:16.176390 3913 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:16.178491 3792 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:16.185833 3885 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240501 14:00:16.235405 3906 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:16.241529 3904 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 251.581 ms
time per row: 0.125791 ms
-----------------------------------------------------
Suppressions used:
count bytes template
16 4220 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:17.453009 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:17.462390 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 623b8a9d77534554ae9fa11f75547a61 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.462908 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623b8a9d77534554ae9fa11f75547a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.463268 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 623b8a9d77534554ae9fa11f75547a61 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.463393 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 96a4b31bcf104c09a3e24d72a3ed9326 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.463694 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623b8a9d77534554ae9fa11f75547a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.463733 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a4b31bcf104c09a3e24d72a3ed9326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.464509 2386 tablet_service.cc:1467] Processing CreateTablet for tablet dd33692bc8c942628f2e403e38a52e99 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.464864 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd33692bc8c942628f2e403e38a52e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.466812 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 96a4b31bcf104c09a3e24d72a3ed9326 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.467203 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a4b31bcf104c09a3e24d72a3ed9326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.463603 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 96a4b31bcf104c09a3e24d72a3ed9326 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.467626 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 623b8a9d77534554ae9fa11f75547a61 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.467825 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a4b31bcf104c09a3e24d72a3ed9326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.468012 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 623b8a9d77534554ae9fa11f75547a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.470862 2126 tablet_service.cc:1467] Processing CreateTablet for tablet dd33692bc8c942628f2e403e38a52e99 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.471261 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd33692bc8c942628f2e403e38a52e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.467626 2256 tablet_service.cc:1467] Processing CreateTablet for tablet dd33692bc8c942628f2e403e38a52e99 (DEFAULT_TABLE table=rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:17.475482 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd33692bc8c942628f2e403e38a52e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:17.498242 3983 tablet_bootstrap.cc:492] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:17.501295 3984 tablet_bootstrap.cc:492] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:17.503185 3982 tablet_bootstrap.cc:492] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:17.504524 3983 tablet_bootstrap.cc:654] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.505244 3982 tablet_bootstrap.cc:654] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.505966 3984 tablet_bootstrap.cc:654] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.507331 3982 tablet_bootstrap.cc:492] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:17.507560 3982 ts_tablet_manager.cc:1367] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:00:17.507671 3980 tablet_bootstrap.cc:492] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:17.508535 3982 raft_consensus.cc:348] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.508713 3982 raft_consensus.cc:374] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.508821 3982 raft_consensus.cc:729] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.509058 3982 consensus_queue.cc:260] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.509477 3980 tablet_bootstrap.cc:654] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.509543 3982 ts_tablet_manager.cc:1398] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:17.512591 3980 tablet_bootstrap.cc:492] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:17.512765 3980 ts_tablet_manager.cc:1367] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:17.512873 3984 tablet_bootstrap.cc:492] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:17.512934 3983 tablet_bootstrap.cc:492] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:17.513026 3984 ts_tablet_manager.cc:1367] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240501 14:00:17.513087 3983 ts_tablet_manager.cc:1367] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240501 14:00:17.513303 3904 raft_consensus.cc:482] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:17.513479 3904 raft_consensus.cc:504] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.513577 3980 raft_consensus.cc:348] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.513769 3980 raft_consensus.cc:374] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.513854 3980 raft_consensus.cc:729] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.513780 3983 raft_consensus.cc:348] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.513964 3983 raft_consensus.cc:374] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.513898 3984 raft_consensus.cc:348] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.514040 3983 raft_consensus.cc:729] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.514082 3984 raft_consensus.cc:374] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.514124 3904 leader_election.cc:288] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:17.514169 3984 raft_consensus.cc:729] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.514081 3980 consensus_queue.cc:260] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.514246 3983 consensus_queue.cc:260] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.514467 3984 consensus_queue.cc:260] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.514632 3980 ts_tablet_manager.cc:1398] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:17.514796 3983 ts_tablet_manager.cc:1398] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:17.514925 3980 tablet_bootstrap.cc:492] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:17.514918 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a4b31bcf104c09a3e24d72a3ed9326" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:17.515051 3983 tablet_bootstrap.cc:492] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:17.515874 3984 ts_tablet_manager.cc:1398] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240501 14:00:17.515959 2471 leader_election.cc:341] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:17.516571 3980 tablet_bootstrap.cc:654] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.516978 3984 tablet_bootstrap.cc:492] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:17.518728 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a4b31bcf104c09a3e24d72a3ed9326" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:17.518985 2145 raft_consensus.cc:2454] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:17.520282 2471 leader_election.cc:302] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters: a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:17.520364 3983 tablet_bootstrap.cc:654] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.521237 3984 tablet_bootstrap.cc:654] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.521411 3904 raft_consensus.cc:2781] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:17.521543 3904 raft_consensus.cc:482] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:17.521652 3904 raft_consensus.cc:3037] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.524780 3980 tablet_bootstrap.cc:492] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:17.524967 3980 ts_tablet_manager.cc:1367] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 14:00:17.525293 3904 raft_consensus.cc:504] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.526228 3980 raft_consensus.cc:348] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.526402 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a4b31bcf104c09a3e24d72a3ed9326" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:17.526522 3980 raft_consensus.cc:374] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.526654 3980 raft_consensus.cc:729] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
W20240501 14:00:17.526988 2471 leader_election.cc:341] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:17.526930 3980 consensus_queue.cc:260] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.527683 3980 ts_tablet_manager.cc:1398] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:17.527609 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a4b31bcf104c09a3e24d72a3ed9326" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:17.527882 2145 raft_consensus.cc:3037] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.528059 3980 tablet_bootstrap.cc:492] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:17.528115 3983 tablet_bootstrap.cc:492] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:17.528282 3983 ts_tablet_manager.cc:1367] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240501 14:00:17.529027 3983 raft_consensus.cc:348] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.529254 3983 raft_consensus.cc:374] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.529331 3983 raft_consensus.cc:729] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.529570 3983 consensus_queue.cc:260] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.529927 3980 tablet_bootstrap.cc:654] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.530098 3983 ts_tablet_manager.cc:1398] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:17.530411 3983 tablet_bootstrap.cc:492] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:17.531190 2145 raft_consensus.cc:2454] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:17.531706 2471 leader_election.cc:302] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters: a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:17.526966 3904 leader_election.cc:288] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:17.532711 3904 raft_consensus.cc:2781] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:17.532862 3904 raft_consensus.cc:686] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:17.533138 3904 consensus_queue.cc:237] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.534129 3983 tablet_bootstrap.cc:654] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:17.543947 3983 tablet_bootstrap.cc:492] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:17.544140 3983 ts_tablet_manager.cc:1367] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20240501 14:00:17.544886 3983 raft_consensus.cc:348] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.545123 3983 raft_consensus.cc:374] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.545238 3983 raft_consensus.cc:729] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.545468 3983 consensus_queue.cc:260] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.546026 3983 ts_tablet_manager.cc:1398] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:17.546082 3980 tablet_bootstrap.cc:492] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:17.546227 3980 ts_tablet_manager.cc:1367] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240501 14:00:17.547030 3980 raft_consensus.cc:348] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.547340 3980 raft_consensus.cc:374] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.547441 3980 raft_consensus.cc:729] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.547453 3984 tablet_bootstrap.cc:492] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:17.547621 3984 ts_tablet_manager.cc:1367] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
I20240501 14:00:17.547643 3980 consensus_queue.cc:260] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.548344 3984 raft_consensus.cc:348] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.548466 3980 ts_tablet_manager.cc:1398] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:17.548553 3984 raft_consensus.cc:374] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:17.548619 3984 raft_consensus.cc:729] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:17.548871 3984 consensus_queue.cc:260] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.549410 3984 ts_tablet_manager.cc:1398] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:17.549201 1999 catalog_manager.cc:5484] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:17.608534 3974 raft_consensus.cc:482] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:17.608752 3974 raft_consensus.cc:504] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.609477 3974 leader_election.cc:288] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:17.609958 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623b8a9d77534554ae9fa11f75547a61" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:17.610039 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623b8a9d77534554ae9fa11f75547a61" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:17.610232 2145 raft_consensus.cc:2454] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:17.610275 2277 raft_consensus.cc:2454] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:17.610725 2341 leader_election.cc:302] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:17.611200 3974 raft_consensus.cc:2781] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:17.611318 3974 raft_consensus.cc:482] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:17.611405 3974 raft_consensus.cc:3037] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.614076 3974 raft_consensus.cc:504] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.615018 3974 leader_election.cc:288] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:17.615276 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623b8a9d77534554ae9fa11f75547a61" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:17.615350 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "623b8a9d77534554ae9fa11f75547a61" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:17.615500 2145 raft_consensus.cc:3037] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.615558 2277 raft_consensus.cc:3037] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.617348 2145 raft_consensus.cc:2454] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:17.617364 2277 raft_consensus.cc:2454] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:17.617736 2341 leader_election.cc:302] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:17.618039 3974 raft_consensus.cc:2781] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:17.618235 3974 raft_consensus.cc:686] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:17.618607 3974 consensus_queue.cc:237] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:17.621659 2004 catalog_manager.cc:5484] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:17.629534 3974 raft_consensus.cc:482] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:17.629774 3974 raft_consensus.cc:504] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:17.630607 3974 leader_election.cc:288] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:17.631088 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a4b31bcf104c09a3e24d72a3ed9326" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:17.631217 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a4b31bcf104c09a3e24d72a3ed9326" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:17.631484 2145 raft_consensus.cc:2379] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64f34785a7b44fa8070b0452a4e0ddd in current term 1: Already voted for candidate 745ac12076e84effa5bde1ecbd6a32ec in this term.
I20240501 14:00:17.632037 2341 leader_election.cc:302] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887
I20240501 14:00:17.632447 3974 raft_consensus.cc:3037] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.634389 3974 raft_consensus.cc:2726] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:17.818354 3847 raft_consensus.cc:482] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:17.818593 3847 raft_consensus.cc:504] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.819252 3847 leader_election.cc:288] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:17.819829 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd33692bc8c942628f2e403e38a52e99" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:17.819849 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd33692bc8c942628f2e403e38a52e99" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:17.820034 2277 raft_consensus.cc:2454] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:17.820067 2406 raft_consensus.cc:2454] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:17.820573 2080 leader_election.cc:302] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:17.820865 3847 raft_consensus.cc:2781] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:17.820972 3847 raft_consensus.cc:482] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:17.821055 3847 raft_consensus.cc:3037] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.823127 3847 raft_consensus.cc:504] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.823716 3847 leader_election.cc:288] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:17.824298 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd33692bc8c942628f2e403e38a52e99" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:17.824314 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd33692bc8c942628f2e403e38a52e99" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:17.824532 2277 raft_consensus.cc:3037] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.824553 2406 raft_consensus.cc:3037] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:17.827354 2406 raft_consensus.cc:2454] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:17.827370 2277 raft_consensus.cc:2454] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:17.827831 2080 leader_election.cc:302] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:17.828171 3847 raft_consensus.cc:2781] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:17.828318 3847 raft_consensus.cc:686] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:17.828583 3847 consensus_queue.cc:237] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:17.831844 2004 catalog_manager.cc:5484] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:17.972601 3992 consensus_queue.cc:1035] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:17.978996 3904 consensus_queue.cc:1035] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:18.177434 3974 consensus_queue.cc:1035] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:18.184181 3974 consensus_queue.cc:1035] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:18.215111 3999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:18.215454 3999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:18.223395 3999 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:18.242398 3847 consensus_queue.cc:1035] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:18.248847 3981 consensus_queue.cc:1035] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
INSERT report
rows total: 2000
time total: 241.643 ms
time per row: 0.120821 ms
I20240501 14:00:18.619446 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:18.628690 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 65306dbef405443fb4c7188ccb762b83 (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.629212 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65306dbef405443fb4c7188ccb762b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.629436 2258 tablet_service.cc:1467] Processing CreateTablet for tablet dcb6951344ad49b3aee27205e898c347 (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.629645 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 65306dbef405443fb4c7188ccb762b83 (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.629845 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb6951344ad49b3aee27205e898c347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.630038 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65306dbef405443fb4c7188ccb762b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.631264 2386 tablet_service.cc:1467] Processing CreateTablet for tablet dcb6951344ad49b3aee27205e898c347 (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.631465 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 387698a371cc42eab970f2f4471bbf5b (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.631698 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb6951344ad49b3aee27205e898c347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.632827 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 387698a371cc42eab970f2f4471bbf5b (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.633107 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387698a371cc42eab970f2f4471bbf5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.633212 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387698a371cc42eab970f2f4471bbf5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.633270 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 387698a371cc42eab970f2f4471bbf5b (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.633662 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387698a371cc42eab970f2f4471bbf5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.634934 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 65306dbef405443fb4c7188ccb762b83 (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.635288 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65306dbef405443fb4c7188ccb762b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.639075 2517 tablet_service.cc:1467] Processing CreateTablet for tablet dcb6951344ad49b3aee27205e898c347 (DEFAULT_TABLE table=rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:18.639461 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb6951344ad49b3aee27205e898c347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:18.647543 4064 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:18.647847 4062 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:18.649629 4064 tablet_bootstrap.cc:654] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.649873 4062 tablet_bootstrap.cc:654] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.665496 4067 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:18.667994 4067 tablet_bootstrap.cc:654] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.669726 4063 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:18.671655 4067 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:18.671890 4067 ts_tablet_manager.cc:1367] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:00:18.672006 4063 tablet_bootstrap.cc:654] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.672573 4064 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:18.672785 4064 ts_tablet_manager.cc:1367] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.015s
I20240501 14:00:18.672837 4067 raft_consensus.cc:348] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.673059 4067 raft_consensus.cc:374] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.673213 4067 raft_consensus.cc:729] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.673535 4067 consensus_queue.cc:260] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.673592 4064 raft_consensus.cc:348] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.673784 4064 raft_consensus.cc:374] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.673887 4064 raft_consensus.cc:729] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.674154 4064 consensus_queue.cc:260] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.674347 4067 ts_tablet_manager.cc:1398] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:18.674860 4067 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:18.674849 4064 ts_tablet_manager.cc:1398] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:18.675242 4064 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:18.676784 4067 tablet_bootstrap.cc:654] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.680050 4062 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:18.680245 4062 ts_tablet_manager.cc:1367] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.017s
I20240501 14:00:18.681144 4062 raft_consensus.cc:348] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.681452 4062 raft_consensus.cc:374] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.681579 4062 raft_consensus.cc:729] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.681931 4064 tablet_bootstrap.cc:654] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.681883 4062 consensus_queue.cc:260] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.682485 4063 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:18.682554 4062 ts_tablet_manager.cc:1398] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:00:18.682713 4063 ts_tablet_manager.cc:1367] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 14:00:18.683487 4062 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:18.683523 4063 raft_consensus.cc:348] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.683763 4063 raft_consensus.cc:374] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.683898 4063 raft_consensus.cc:729] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.684160 4067 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:18.684201 4063 consensus_queue.cc:260] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.685837 4063 ts_tablet_manager.cc:1398] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:18.686215 4063 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:18.686235 4067 ts_tablet_manager.cc:1367] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 14:00:18.687093 4067 raft_consensus.cc:348] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.687393 4067 raft_consensus.cc:374] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.687516 4067 raft_consensus.cc:729] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.687831 4067 consensus_queue.cc:260] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.689694 4067 ts_tablet_manager.cc:1398] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:18.689873 4062 tablet_bootstrap.cc:654] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.690095 4067 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:18.691045 4063 tablet_bootstrap.cc:654] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.692176 4067 tablet_bootstrap.cc:654] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:18.693032 4063 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:18.693733 4063 ts_tablet_manager.cc:1367] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:18.694681 4063 raft_consensus.cc:348] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.695184 4063 raft_consensus.cc:374] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.695740 4063 raft_consensus.cc:729] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.696075 4063 consensus_queue.cc:260] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.698222 4063 ts_tablet_manager.cc:1398] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:18.700764 4067 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:18.700984 4067 ts_tablet_manager.cc:1367] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20240501 14:00:18.701828 4067 raft_consensus.cc:348] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.702069 4067 raft_consensus.cc:374] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.702169 4067 raft_consensus.cc:729] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.702437 4067 consensus_queue.cc:260] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.703186 4064 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:18.703586 4064 ts_tablet_manager.cc:1367] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.004s
I20240501 14:00:18.704635 4064 raft_consensus.cc:348] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.704818 4064 raft_consensus.cc:374] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.704913 4064 raft_consensus.cc:729] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.705142 4064 consensus_queue.cc:260] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.705766 4064 ts_tablet_manager.cc:1398] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:18.706480 4067 ts_tablet_manager.cc:1398] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:18.707172 4062 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:18.707350 4062 ts_tablet_manager.cc:1367] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.009s
I20240501 14:00:18.708132 4062 raft_consensus.cc:348] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.708375 4062 raft_consensus.cc:374] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:18.708457 4062 raft_consensus.cc:729] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:18.708725 4062 consensus_queue.cc:260] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.709355 4062 ts_tablet_manager.cc:1398] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:00:18.721518 3949 raft_consensus.cc:482] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.721726 3949 raft_consensus.cc:504] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.722425 3949 leader_election.cc:288] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:18.722960 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65306dbef405443fb4c7188ccb762b83" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:18.723007 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65306dbef405443fb4c7188ccb762b83" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:18.723188 2146 raft_consensus.cc:2454] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:18.723237 2537 raft_consensus.cc:2454] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:18.723655 2211 leader_election.cc:302] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:18.723914 3949 raft_consensus.cc:2781] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.724036 3949 raft_consensus.cc:482] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.724126 3949 raft_consensus.cc:3037] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.726037 3949 raft_consensus.cc:504] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.726727 3949 leader_election.cc:288] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:18.727222 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65306dbef405443fb4c7188ccb762b83" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:18.727249 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65306dbef405443fb4c7188ccb762b83" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:18.727421 2146 raft_consensus.cc:3037] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.727463 2537 raft_consensus.cc:3037] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.729488 2537 raft_consensus.cc:2454] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:18.729511 2146 raft_consensus.cc:2454] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:18.729897 2211 leader_election.cc:302] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:18.730180 3949 raft_consensus.cc:2781] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.730610 3949 raft_consensus.cc:686] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:18.730875 3949 consensus_queue.cc:237] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:18.737321 2004 catalog_manager.cc:5484] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:18.822448 3974 raft_consensus.cc:482] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.822683 3974 raft_consensus.cc:504] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.823374 3974 leader_election.cc:288] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:18.823807 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb6951344ad49b3aee27205e898c347" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:18.823882 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb6951344ad49b3aee27205e898c347" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:18.824031 2537 raft_consensus.cc:2454] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:18.824138 2277 raft_consensus.cc:2454] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:18.824460 2341 leader_election.cc:302] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:18.824704 3974 raft_consensus.cc:2781] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.824821 3974 raft_consensus.cc:482] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.824908 3974 raft_consensus.cc:3037] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.827913 3974 raft_consensus.cc:504] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.828503 3974 leader_election.cc:288] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:18.829010 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb6951344ad49b3aee27205e898c347" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:18.829084 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb6951344ad49b3aee27205e898c347" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:18.829231 2537 raft_consensus.cc:3037] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.829303 2277 raft_consensus.cc:3037] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.830705 3847 raft_consensus.cc:482] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:18.830874 3847 raft_consensus.cc:504] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.831637 2277 raft_consensus.cc:2454] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:18.831650 3847 leader_election.cc:288] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:18.831943 2537 raft_consensus.cc:2454] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:18.832208 2341 leader_election.cc:302] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:18.832201 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387698a371cc42eab970f2f4471bbf5b" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:18.832413 2536 raft_consensus.cc:2454] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:18.832600 3974 raft_consensus.cc:2781] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.832758 3974 raft_consensus.cc:686] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:18.833088 3974 consensus_queue.cc:237] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:18.833257 2080 leader_election.cc:302] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:18.833251 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387698a371cc42eab970f2f4471bbf5b" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:18.833472 2408 raft_consensus.cc:2454] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:18.833537 3847 raft_consensus.cc:2781] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:18.833643 3847 raft_consensus.cc:482] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:18.833729 3847 raft_consensus.cc:3037] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.835867 3847 raft_consensus.cc:504] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.836447 3847 leader_election.cc:288] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:18.836535 2004 catalog_manager.cc:5484] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:18.836834 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387698a371cc42eab970f2f4471bbf5b" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:18.837049 2408 raft_consensus.cc:3037] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.836992 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387698a371cc42eab970f2f4471bbf5b" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:18.837211 2536 raft_consensus.cc:3037] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:18.839762 2408 raft_consensus.cc:2454] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:18.839799 2536 raft_consensus.cc:2454] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:18.840159 2080 leader_election.cc:302] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:18.840463 3847 raft_consensus.cc:2781] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:18.840605 3847 raft_consensus.cc:686] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:18.840881 3847 consensus_queue.cc:237] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:18.844022 2004 catalog_manager.cc:5484] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:19.176189 3949 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:19.182461 3949 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:19.242265 4079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:19.242666 4079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:19.250458 4079 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 14:00:19.340329 3847 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:19.390977 4110 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:19.411733 3974 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:19.499037 4132 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20240501 14:00:19.543490 2341 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:41309, user_credentials={real_user=slave}} blocked reactor thread for 43881.8us
INSERT report
rows total: 2000
time total: 306.385 ms
time per row: 0.153192 ms
I20240501 14:00:19.704674 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:19.714110 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 47fda0a6eca040779405bcdb44a15d13 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.714649 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fda0a6eca040779405bcdb44a15d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.714551 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 47fda0a6eca040779405bcdb44a15d13 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.714848 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 32164bd2be9a49eabc787ab9085db994 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.715024 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fda0a6eca040779405bcdb44a15d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.715246 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32164bd2be9a49eabc787ab9085db994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.718734 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 47fda0a6eca040779405bcdb44a15d13 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.719166 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47fda0a6eca040779405bcdb44a15d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.723542 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 32164bd2be9a49eabc787ab9085db994 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.723680 2257 tablet_service.cc:1467] Processing CreateTablet for tablet e2d270a4918a46c08f699272101cb0fc (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.724005 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32164bd2be9a49eabc787ab9085db994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.725090 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 32164bd2be9a49eabc787ab9085db994 (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.725492 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32164bd2be9a49eabc787ab9085db994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.729502 4148 tablet_bootstrap.cc:492] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:19.729936 4149 tablet_bootstrap.cc:492] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:19.729856 2388 tablet_service.cc:1467] Processing CreateTablet for tablet e2d270a4918a46c08f699272101cb0fc (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.730321 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d270a4918a46c08f699272101cb0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.731217 2518 tablet_service.cc:1467] Processing CreateTablet for tablet e2d270a4918a46c08f699272101cb0fc (DEFAULT_TABLE table=rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:19.731655 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d270a4918a46c08f699272101cb0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.731647 4148 tablet_bootstrap.cc:654] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.733326 4149 tablet_bootstrap.cc:654] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.738675 4150 tablet_bootstrap.cc:492] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:19.738926 4149 tablet_bootstrap.cc:492] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:19.739096 4149 ts_tablet_manager.cc:1367] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:00:19.739504 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d270a4918a46c08f699272101cb0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:19.740442 4150 tablet_bootstrap.cc:654] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.740391 4149 raft_consensus.cc:348] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.740634 4149 raft_consensus.cc:374] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.740728 4149 raft_consensus.cc:729] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.740957 4149 consensus_queue.cc:260] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.744015 4149 ts_tablet_manager.cc:1398] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:19.744330 4149 tablet_bootstrap.cc:492] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:19.746853 4149 tablet_bootstrap.cc:654] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.750810 4150 tablet_bootstrap.cc:492] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:19.751458 4149 tablet_bootstrap.cc:492] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:19.751624 4149 ts_tablet_manager.cc:1367] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:19.751611 4150 ts_tablet_manager.cc:1367] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20240501 14:00:19.753773 4149 raft_consensus.cc:348] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.754374 4150 raft_consensus.cc:348] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.754587 4149 raft_consensus.cc:374] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.754743 4149 raft_consensus.cc:729] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.755064 4150 raft_consensus.cc:374] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.755064 4149 consensus_queue.cc:260] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.755563 4153 tablet_bootstrap.cc:492] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:19.755851 4149 ts_tablet_manager.cc:1398] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:19.756218 4149 tablet_bootstrap.cc:492] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:19.756701 4150 raft_consensus.cc:729] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.757040 4150 consensus_queue.cc:260] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.758193 4149 tablet_bootstrap.cc:654] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.759996 4150 ts_tablet_manager.cc:1398] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:19.760347 4150 tablet_bootstrap.cc:492] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:19.761966 4153 tablet_bootstrap.cc:654] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.762462 4150 tablet_bootstrap.cc:654] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.763859 4148 tablet_bootstrap.cc:492] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:19.765054 4148 ts_tablet_manager.cc:1367] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.008s
I20240501 14:00:19.766018 4148 raft_consensus.cc:348] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.766333 4148 raft_consensus.cc:374] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.766484 4148 raft_consensus.cc:729] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.766911 4148 consensus_queue.cc:260] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.767575 4148 ts_tablet_manager.cc:1398] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:19.768378 4148 tablet_bootstrap.cc:492] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:19.768883 4150 tablet_bootstrap.cc:492] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:19.769059 4150 ts_tablet_manager.cc:1367] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:00:19.769862 4150 raft_consensus.cc:348] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:19.770077 4150 raft_consensus.cc:374] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.770148 4150 raft_consensus.cc:729] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.770473 4150 consensus_queue.cc:260] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:19.770916 4149 tablet_bootstrap.cc:492] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:19.771066 4149 ts_tablet_manager.cc:1367] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240501 14:00:19.771253 4148 tablet_bootstrap.cc:654] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.772099 4150 ts_tablet_manager.cc:1398] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:19.772900 4153 tablet_bootstrap.cc:492] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:19.773102 4153 ts_tablet_manager.cc:1367] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240501 14:00:19.773387 4148 tablet_bootstrap.cc:492] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:19.773548 4148 ts_tablet_manager.cc:1367] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 14:00:19.773476 4149 raft_consensus.cc:348] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:19.773747 4149 raft_consensus.cc:374] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.773895 4149 raft_consensus.cc:729] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.774147 4149 consensus_queue.cc:260] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:19.774261 4153 raft_consensus.cc:348] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:19.774375 4148 raft_consensus.cc:348] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.774857 4148 raft_consensus.cc:374] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.774928 4149 ts_tablet_manager.cc:1398] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:19.774968 4148 raft_consensus.cc:729] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.774607 4153 raft_consensus.cc:374] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.775425 4153 raft_consensus.cc:729] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.775741 4153 consensus_queue.cc:260] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:19.775245 4148 consensus_queue.cc:260] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.776528 4148 ts_tablet_manager.cc:1398] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:19.777485 4153 ts_tablet_manager.cc:1398] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:19.777829 4153 tablet_bootstrap.cc:492] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:19.781466 4153 tablet_bootstrap.cc:654] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:19.783546 4153 tablet_bootstrap.cc:492] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:19.783721 4153 ts_tablet_manager.cc:1367] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:00:19.784387 4153 raft_consensus.cc:348] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.784616 4153 raft_consensus.cc:374] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:19.784698 4153 raft_consensus.cc:729] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:19.784904 4153 consensus_queue.cc:260] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.785352 4153 ts_tablet_manager.cc:1398] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:19.846670 4132 raft_consensus.cc:482] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.846868 4132 raft_consensus.cc:504] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.847718 4132 leader_election.cc:288] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:19.848228 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32164bd2be9a49eabc787ab9085db994" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:19.848240 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32164bd2be9a49eabc787ab9085db994" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:19.848456 2145 raft_consensus.cc:2454] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:19.848475 2277 raft_consensus.cc:2454] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:19.848904 2341 leader_election.cc:302] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:19.849210 4132 raft_consensus.cc:2781] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.849313 4132 raft_consensus.cc:482] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.849424 4132 raft_consensus.cc:3037] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.851444 4132 raft_consensus.cc:504] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.852032 4132 leader_election.cc:288] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:19.852428 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32164bd2be9a49eabc787ab9085db994" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:19.852516 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32164bd2be9a49eabc787ab9085db994" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:19.852674 2277 raft_consensus.cc:3037] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.852682 2145 raft_consensus.cc:3037] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.854573 2277 raft_consensus.cc:2454] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:19.854570 2145 raft_consensus.cc:2454] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:19.855002 2341 leader_election.cc:302] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:19.855268 4132 raft_consensus.cc:2781] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:19.855448 4132 raft_consensus.cc:686] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:19.855712 4132 consensus_queue.cc:237] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:19.859273 2002 catalog_manager.cc:5484] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:19.934103 4080 raft_consensus.cc:482] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:19.934305 4080 raft_consensus.cc:504] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.935194 4080 leader_election.cc:288] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:19.935745 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fda0a6eca040779405bcdb44a15d13" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:19.935882 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fda0a6eca040779405bcdb44a15d13" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:19.935999 2145 raft_consensus.cc:2454] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:19.936098 2277 raft_consensus.cc:2454] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:19.936440 2471 leader_election.cc:302] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:19.936730 4080 raft_consensus.cc:2781] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:19.936831 4080 raft_consensus.cc:482] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:19.936908 4080 raft_consensus.cc:3037] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.939116 4080 raft_consensus.cc:504] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.939668 4080 leader_election.cc:288] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:19.940207 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fda0a6eca040779405bcdb44a15d13" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:19.940219 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47fda0a6eca040779405bcdb44a15d13" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:19.940384 2277 raft_consensus.cc:3037] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.940408 2145 raft_consensus.cc:3037] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:19.942175 2145 raft_consensus.cc:2454] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:19.942206 2277 raft_consensus.cc:2454] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:19.942631 2471 leader_election.cc:302] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:19.942898 4080 raft_consensus.cc:2781] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:19.943264 4080 raft_consensus.cc:686] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:19.943563 4080 consensus_queue.cc:237] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:19.946615 2002 catalog_manager.cc:5484] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:20.124150 4080 raft_consensus.cc:482] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:20.124341 4080 raft_consensus.cc:504] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:20.125030 4080 leader_election.cc:288] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:20.125540 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d270a4918a46c08f699272101cb0fc" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:20.125540 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d270a4918a46c08f699272101cb0fc" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:20.125761 2277 raft_consensus.cc:2454] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:20.125764 2406 raft_consensus.cc:2454] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:20.126210 2471 leader_election.cc:302] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:20.126528 4080 raft_consensus.cc:2781] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:20.126649 4080 raft_consensus.cc:482] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:20.126925 4080 raft_consensus.cc:3037] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.128756 4080 raft_consensus.cc:504] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:20.129277 4080 leader_election.cc:288] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:20.129703 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d270a4918a46c08f699272101cb0fc" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:20.129712 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d270a4918a46c08f699272101cb0fc" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:20.129868 2406 raft_consensus.cc:3037] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.129917 2277 raft_consensus.cc:3037] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.132150 2277 raft_consensus.cc:2454] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:20.132182 2406 raft_consensus.cc:2454] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:20.132534 2471 leader_election.cc:302] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:20.132781 4080 raft_consensus.cc:2781] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:20.132925 4080 raft_consensus.cc:686] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:20.133198 4080 consensus_queue.cc:237] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:20.136106 2002 catalog_manager.cc:5484] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:20.341161 4135 consensus_queue.cc:1035] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.354343 4132 consensus_queue.cc:1035] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.384924 4080 consensus_queue.cc:1035] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.391130 4162 consensus_queue.cc:1035] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.514535 4162 consensus_queue.cc:1035] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:20.520403 4080 consensus_queue.cc:1035] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:20.570281 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:20.570694 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:20.578812 4165 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
INSERT report
rows total: 2000
time total: 248.924 ms
time per row: 0.124462 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:21.874044 2006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:21.883424 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 457c62e1ddf547ad900ed529b3244474 (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.883841 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457c62e1ddf547ad900ed529b3244474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.885555 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 457c62e1ddf547ad900ed529b3244474 (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.885761 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 457c62e1ddf547ad900ed529b3244474 (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.885986 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457c62e1ddf547ad900ed529b3244474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.886149 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457c62e1ddf547ad900ed529b3244474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.887370 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 238aa15a2cd74ff69e17a2934b32b70d (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.887764 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238aa15a2cd74ff69e17a2934b32b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.888935 2517 tablet_service.cc:1467] Processing CreateTablet for tablet f8ae4d9e24a94b878627722ad94c0e35 (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.889294 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ae4d9e24a94b878627722ad94c0e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.890866 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 238aa15a2cd74ff69e17a2934b32b70d (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.890878 2386 tablet_service.cc:1467] Processing CreateTablet for tablet f8ae4d9e24a94b878627722ad94c0e35 (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.891237 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ae4d9e24a94b878627722ad94c0e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.891239 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238aa15a2cd74ff69e17a2934b32b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.894575 2126 tablet_service.cc:1467] Processing CreateTablet for tablet f8ae4d9e24a94b878627722ad94c0e35 (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.894899 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 238aa15a2cd74ff69e17a2934b32b70d (DEFAULT_TABLE table=rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.894999 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ae4d9e24a94b878627722ad94c0e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.895295 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238aa15a2cd74ff69e17a2934b32b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.911772 4241 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:21.913740 4241 tablet_bootstrap.cc:654] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.915789 4239 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:21.917778 4239 tablet_bootstrap.cc:654] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.918152 4242 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:21.919797 4242 tablet_bootstrap.cc:654] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.920145 4241 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:21.920377 4241 ts_tablet_manager.cc:1367] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 14:00:21.921394 4241 raft_consensus.cc:348] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.921840 4238 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:21.921929 4241 raft_consensus.cc:374] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.922091 4241 raft_consensus.cc:729] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.922871 4239 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:21.923064 4239 ts_tablet_manager.cc:1367] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240501 14:00:21.923707 4238 tablet_bootstrap.cc:654] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.923853 4239 raft_consensus.cc:348] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.924096 4239 raft_consensus.cc:374] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.924223 4239 raft_consensus.cc:729] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.924476 4239 consensus_queue.cc:260] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.925222 4239 ts_tablet_manager.cc:1398] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:21.925513 4242 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:21.925670 4242 ts_tablet_manager.cc:1367] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240501 14:00:21.922533 4241 consensus_queue.cc:260] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.926793 4239 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:21.926685 4242 raft_consensus.cc:348] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.926882 4242 raft_consensus.cc:374] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.926966 4242 raft_consensus.cc:729] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.927192 4242 consensus_queue.cc:260] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.927707 4242 ts_tablet_manager.cc:1398] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:21.928290 4241 ts_tablet_manager.cc:1398] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:21.928972 4242 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:21.930589 4242 tablet_bootstrap.cc:654] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.932179 4238 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:21.932397 4238 ts_tablet_manager.cc:1367] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 14:00:21.933346 4238 raft_consensus.cc:348] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:21.933555 4239 tablet_bootstrap.cc:654] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.933641 4238 raft_consensus.cc:374] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.933797 4238 raft_consensus.cc:729] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.934099 4238 consensus_queue.cc:260] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:21.934913 4238 ts_tablet_manager.cc:1398] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:21.935263 4238 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:21.936915 4238 tablet_bootstrap.cc:654] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.938323 4242 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:21.938531 4242 ts_tablet_manager.cc:1367] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240501 14:00:21.939285 4242 raft_consensus.cc:348] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:21.939518 4242 raft_consensus.cc:374] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.939606 4242 raft_consensus.cc:729] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.939853 4242 consensus_queue.cc:260] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:21.940446 4242 ts_tablet_manager.cc:1398] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:21.943975 4239 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:21.944162 4239 ts_tablet_manager.cc:1367] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:00:21.944845 4239 raft_consensus.cc:348] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.945065 4239 raft_consensus.cc:374] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.945143 4239 raft_consensus.cc:729] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.945390 4239 consensus_queue.cc:260] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.945570 4238 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:21.945757 4238 ts_tablet_manager.cc:1367] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240501 14:00:21.946003 4239 ts_tablet_manager.cc:1398] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:21.946384 4239 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:21.946338 4238 raft_consensus.cc:348] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.946584 4238 raft_consensus.cc:374] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.946656 4238 raft_consensus.cc:729] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.946919 4238 consensus_queue.cc:260] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.947501 4238 ts_tablet_manager.cc:1398] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:21.948611 4238 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:21.948698 4239 tablet_bootstrap.cc:654] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.950489 4238 tablet_bootstrap.cc:654] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.957335 4239 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:21.957479 4239 ts_tablet_manager.cc:1367] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240501 14:00:21.957656 4238 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:21.957799 4238 ts_tablet_manager.cc:1367] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240501 14:00:21.958165 4239 raft_consensus.cc:348] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:21.958328 4239 raft_consensus.cc:374] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.958408 4239 raft_consensus.cc:729] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.958442 4238 raft_consensus.cc:348] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.958655 4238 raft_consensus.cc:374] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.958614 4239 consensus_queue.cc:260] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:21.958743 4238 raft_consensus.cc:729] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.958954 4238 consensus_queue.cc:260] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:21.959167 4239 ts_tablet_manager.cc:1398] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:21.959573 4238 ts_tablet_manager.cc:1398] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:22.022363 4233 raft_consensus.cc:482] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.022589 4233 raft_consensus.cc:504] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:22.023329 4233 leader_election.cc:288] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:22.023800 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457c62e1ddf547ad900ed529b3244474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:22.023823 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457c62e1ddf547ad900ed529b3244474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:22.024070 2147 raft_consensus.cc:2454] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:22.024063 2537 raft_consensus.cc:2454] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:22.024533 2341 leader_election.cc:302] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:22.024832 4233 raft_consensus.cc:2781] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.024957 4233 raft_consensus.cc:482] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.025034 4233 raft_consensus.cc:3037] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.027029 4233 raft_consensus.cc:504] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:22.027618 4233 leader_election.cc:288] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:22.028096 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457c62e1ddf547ad900ed529b3244474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:22.028331 2537 raft_consensus.cc:3037] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.028432 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457c62e1ddf547ad900ed529b3244474" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:22.028638 2147 raft_consensus.cc:3037] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.031131 2537 raft_consensus.cc:2454] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:22.031162 2147 raft_consensus.cc:2454] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:22.031548 2341 leader_election.cc:302] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:22.031853 4233 raft_consensus.cc:2781] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.031980 4233 raft_consensus.cc:686] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:22.032451 4233 consensus_queue.cc:237] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:22.035588 1999 catalog_manager.cc:5484] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:22.102774 4080 raft_consensus.cc:482] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.102995 4080 raft_consensus.cc:504] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:22.103678 4080 leader_election.cc:288] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:22.104238 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae4d9e24a94b878627722ad94c0e35" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:22.104295 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae4d9e24a94b878627722ad94c0e35" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:22.104465 2406 raft_consensus.cc:2454] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:22.104519 2147 raft_consensus.cc:2454] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:22.104902 2471 leader_election.cc:302] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:22.105211 4080 raft_consensus.cc:2781] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.105310 4080 raft_consensus.cc:482] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.105394 4080 raft_consensus.cc:3037] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.107653 4080 raft_consensus.cc:504] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:22.108227 4080 leader_election.cc:288] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:22.108565 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae4d9e24a94b878627722ad94c0e35" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:22.108655 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ae4d9e24a94b878627722ad94c0e35" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:22.108772 2406 raft_consensus.cc:3037] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.108851 2147 raft_consensus.cc:3037] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.110743 2406 raft_consensus.cc:2454] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:22.110921 2147 raft_consensus.cc:2454] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:22.111155 2471 leader_election.cc:302] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:22.111459 4080 raft_consensus.cc:2781] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.111609 4080 raft_consensus.cc:686] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:22.111891 4080 consensus_queue.cc:237] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:22.115445 1999 catalog_manager.cc:5484] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:22.192127 4240 raft_consensus.cc:482] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.192296 4240 raft_consensus.cc:504] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:22.193158 4240 leader_election.cc:288] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:22.193609 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238aa15a2cd74ff69e17a2934b32b70d" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:22.193717 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238aa15a2cd74ff69e17a2934b32b70d" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:22.193838 2147 raft_consensus.cc:2454] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:22.193976 2537 raft_consensus.cc:2454] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:22.194286 2211 leader_election.cc:302] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:22.194567 4240 raft_consensus.cc:2781] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.194689 4240 raft_consensus.cc:482] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.194784 4240 raft_consensus.cc:3037] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.196995 4240 raft_consensus.cc:504] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:22.197618 4240 leader_election.cc:288] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:22.198113 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238aa15a2cd74ff69e17a2934b32b70d" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:22.198310 2147 raft_consensus.cc:3037] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.198182 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238aa15a2cd74ff69e17a2934b32b70d" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:22.198547 2537 raft_consensus.cc:3037] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.200484 2147 raft_consensus.cc:2454] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:22.200924 2211 leader_election.cc:302] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:22.201022 2537 raft_consensus.cc:2454] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:22.201342 4240 raft_consensus.cc:2781] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.201490 4240 raft_consensus.cc:686] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:22.201719 4240 consensus_queue.cc:237] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:22.204885 1999 catalog_manager.cc:5484] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:22.498831 4080 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.505093 4232 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.506790 4211 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.523880 4233 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:22.573057 4255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:22.573397 4255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:22.581009 4255 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:22.589551 4176 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.596801 4240 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 293.862 ms
time per row: 0.146931 ms
I20240501 14:00:23.042184 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:23.051801 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 3682f3837f6e40fbbe32e47d948d1305 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.052232 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3682f3837f6e40fbbe32e47d948d1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.052150 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 3682f3837f6e40fbbe32e47d948d1305 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.052620 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3682f3837f6e40fbbe32e47d948d1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.053884 2386 tablet_service.cc:1467] Processing CreateTablet for tablet bc35182173f2418e91a852adc391f5a5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.054044 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 8756f19007eb4704a80213dd469874a7 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.054306 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc35182173f2418e91a852adc391f5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.054466 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8756f19007eb4704a80213dd469874a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.055150 2517 tablet_service.cc:1467] Processing CreateTablet for tablet bc35182173f2418e91a852adc391f5a5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.055538 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc35182173f2418e91a852adc391f5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.058941 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 8756f19007eb4704a80213dd469874a7 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.058923 2258 tablet_service.cc:1467] Processing CreateTablet for tablet bc35182173f2418e91a852adc391f5a5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.059334 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8756f19007eb4704a80213dd469874a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.060662 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc35182173f2418e91a852adc391f5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.063411 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 8756f19007eb4704a80213dd469874a7 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.063832 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8756f19007eb4704a80213dd469874a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.066598 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 3682f3837f6e40fbbe32e47d948d1305 (DEFAULT_TABLE table=rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:23.067034 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3682f3837f6e40fbbe32e47d948d1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:23.073535 4326 tablet_bootstrap.cc:492] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:23.075071 4326 tablet_bootstrap.cc:654] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.077354 4329 tablet_bootstrap.cc:492] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:23.077647 4326 tablet_bootstrap.cc:492] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:23.077881 4326 ts_tablet_manager.cc:1367] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:00:23.078810 4326 raft_consensus.cc:348] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.079200 4326 raft_consensus.cc:374] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.079358 4329 tablet_bootstrap.cc:654] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.079357 4326 raft_consensus.cc:729] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.079866 4326 consensus_queue.cc:260] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.080577 4326 ts_tablet_manager.cc:1398] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:23.081161 4326 tablet_bootstrap.cc:492] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:23.082891 4326 tablet_bootstrap.cc:654] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.092175 4326 tablet_bootstrap.cc:492] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:23.092295 4330 tablet_bootstrap.cc:492] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:23.092626 4326 ts_tablet_manager.cc:1367] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 14:00:23.093557 4326 raft_consensus.cc:348] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.093904 4326 raft_consensus.cc:374] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.094056 4326 raft_consensus.cc:729] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.094429 4326 consensus_queue.cc:260] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.095606 4326 ts_tablet_manager.cc:1398] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:23.096007 4326 tablet_bootstrap.cc:492] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:23.100260 4326 tablet_bootstrap.cc:654] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.103677 4330 tablet_bootstrap.cc:654] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.105914 4327 tablet_bootstrap.cc:492] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:23.106415 4329 tablet_bootstrap.cc:492] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:23.106570 4326 tablet_bootstrap.cc:492] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:23.106621 4329 ts_tablet_manager.cc:1367] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.015s
I20240501 14:00:23.106715 4326 ts_tablet_manager.cc:1367] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20240501 14:00:23.107424 4327 tablet_bootstrap.cc:654] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.107483 4329 raft_consensus.cc:348] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.107483 4326 raft_consensus.cc:348] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.107697 4326 raft_consensus.cc:374] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.107697 4329 raft_consensus.cc:374] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.107790 4329 raft_consensus.cc:729] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.107823 4326 raft_consensus.cc:729] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.108033 4329 consensus_queue.cc:260] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.108084 4326 consensus_queue.cc:260] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.108669 4326 ts_tablet_manager.cc:1398] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:23.108876 4329 ts_tablet_manager.cc:1398] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:23.109198 4329 tablet_bootstrap.cc:492] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:23.113658 4329 tablet_bootstrap.cc:654] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.115124 4330 tablet_bootstrap.cc:492] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:23.115298 4330 ts_tablet_manager.cc:1367] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240501 14:00:23.115937 4330 raft_consensus.cc:348] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.116096 4330 raft_consensus.cc:374] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.116194 4330 raft_consensus.cc:729] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.116432 4330 consensus_queue.cc:260] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.116930 4330 ts_tablet_manager.cc:1398] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:23.117879 4327 tablet_bootstrap.cc:492] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:23.118067 4327 ts_tablet_manager.cc:1367] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240501 14:00:23.118893 4327 raft_consensus.cc:348] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.119107 4327 raft_consensus.cc:374] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.119244 4327 raft_consensus.cc:729] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.119541 4327 consensus_queue.cc:260] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.120170 4327 ts_tablet_manager.cc:1398] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:23.120527 4327 tablet_bootstrap.cc:492] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:23.122202 4327 tablet_bootstrap.cc:654] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.123018 4329 tablet_bootstrap.cc:492] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:23.123221 4329 ts_tablet_manager.cc:1367] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240501 14:00:23.124042 4329 raft_consensus.cc:348] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.124300 4329 raft_consensus.cc:374] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.124418 4329 raft_consensus.cc:729] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.124791 4329 consensus_queue.cc:260] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.125550 4329 ts_tablet_manager.cc:1398] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:23.131493 4327 tablet_bootstrap.cc:492] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:23.131668 4327 ts_tablet_manager.cc:1367] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240501 14:00:23.132316 4327 raft_consensus.cc:348] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.132536 4327 raft_consensus.cc:374] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.132612 4327 raft_consensus.cc:729] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.132805 4327 consensus_queue.cc:260] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.133263 4327 ts_tablet_manager.cc:1398] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:23.133615 4327 tablet_bootstrap.cc:492] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:23.135114 4327 tablet_bootstrap.cc:654] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:23.137842 4327 tablet_bootstrap.cc:492] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:23.138000 4327 ts_tablet_manager.cc:1367] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:23.138818 4327 raft_consensus.cc:348] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.139029 4327 raft_consensus.cc:374] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:23.139140 4327 raft_consensus.cc:729] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:23.139369 4327 consensus_queue.cc:260] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.139825 4327 ts_tablet_manager.cc:1398] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:23.163041 4263 raft_consensus.cc:482] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.163236 4263 raft_consensus.cc:504] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.163930 4263 leader_election.cc:288] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:23.164502 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35182173f2418e91a852adc391f5a5" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:23.164561 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35182173f2418e91a852adc391f5a5" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:23.164719 2277 raft_consensus.cc:2454] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:23.164789 2536 raft_consensus.cc:2454] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:23.165378 2341 leader_election.cc:302] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:23.165627 4263 raft_consensus.cc:2781] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.165763 4263 raft_consensus.cc:482] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.165867 4263 raft_consensus.cc:3037] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.168083 4263 raft_consensus.cc:504] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.168460 4307 raft_consensus.cc:482] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.168628 4307 raft_consensus.cc:504] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.168881 4263 leader_election.cc:288] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:23.169425 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35182173f2418e91a852adc391f5a5" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:23.169425 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35182173f2418e91a852adc391f5a5" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:23.169620 2536 raft_consensus.cc:3037] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.169638 2277 raft_consensus.cc:3037] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.169665 4307 leader_election.cc:288] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:23.170181 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8756f19007eb4704a80213dd469874a7" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:23.170387 2147 raft_consensus.cc:2454] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:23.170367 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8756f19007eb4704a80213dd469874a7" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:23.170574 2406 raft_consensus.cc:2454] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:23.171070 2211 leader_election.cc:302] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:23.171389 4307 raft_consensus.cc:2781] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.171500 4307 raft_consensus.cc:482] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.171576 4307 raft_consensus.cc:3037] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.172098 2536 raft_consensus.cc:2454] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:23.172817 2277 raft_consensus.cc:2454] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:23.173269 2341 leader_election.cc:302] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:23.173600 4263 raft_consensus.cc:2781] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.173763 4263 raft_consensus.cc:686] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:23.174117 4263 consensus_queue.cc:237] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:23.175132 4307 raft_consensus.cc:504] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.175720 4307 leader_election.cc:288] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:23.176378 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8756f19007eb4704a80213dd469874a7" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:23.176442 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8756f19007eb4704a80213dd469874a7" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:23.176592 2406 raft_consensus.cc:3037] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.176627 2147 raft_consensus.cc:3037] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.178681 2406 raft_consensus.cc:2454] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:23.178617 1997 catalog_manager.cc:5484] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:23.179060 2147 raft_consensus.cc:2454] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:23.179076 2211 leader_election.cc:302] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:23.179441 4307 raft_consensus.cc:2781] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.179617 4307 raft_consensus.cc:686] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:23.179894 4307 consensus_queue.cc:237] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.183454 1997 catalog_manager.cc:5484] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:23.251579 4256 raft_consensus.cc:482] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:23.251765 4256 raft_consensus.cc:504] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.252736 4256 leader_election.cc:288] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:23.253314 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3682f3837f6e40fbbe32e47d948d1305" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:23.253310 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3682f3837f6e40fbbe32e47d948d1305" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:23.253592 2277 raft_consensus.cc:2454] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:23.253656 2406 raft_consensus.cc:2454] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:23.254041 2080 leader_election.cc:302] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:23.254357 4256 raft_consensus.cc:2781] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:23.254526 4256 raft_consensus.cc:482] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:23.254639 4256 raft_consensus.cc:3037] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.256608 4256 raft_consensus.cc:504] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.257143 4256 leader_election.cc:288] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:23.257834 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3682f3837f6e40fbbe32e47d948d1305" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:23.257846 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3682f3837f6e40fbbe32e47d948d1305" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:23.258054 2406 raft_consensus.cc:3037] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.258059 2277 raft_consensus.cc:3037] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:23.260062 2406 raft_consensus.cc:2454] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:23.260063 2277 raft_consensus.cc:2454] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:23.260519 2080 leader_election.cc:302] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:23.260871 4256 raft_consensus.cc:2781] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:23.261384 4256 raft_consensus.cc:686] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:23.261757 4256 consensus_queue.cc:237] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:23.266300 1997 catalog_manager.cc:5484] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:23.639587 4263 consensus_queue.cc:1035] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:23.645953 4263 consensus_queue.cc:1035] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:23.693928 4343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:23.694272 4343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:00:23.702306 4307 consensus_queue.cc:1035] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:23.702880 4343 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:23.710673 4307 consensus_queue.cc:1035] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
I20240501 14:00:23.879065 4256 consensus_queue.cc:1035] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:23.900049 4256 consensus_queue.cc:1035] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:23.935422 2341 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.4:36117, user_credentials={real_user=slave}} blocked reactor thread for 45410.4us
I20240501 14:00:23.964910 4385 mvcc.cc:205] Tried to move back new op lower bound from 7022887009803857920 to 7022887007283228672. Current Snapshot: MvccSnapshot[applied={T|T < 7022887009803857920}]
I20240501 14:00:23.984010 4346 mvcc.cc:205] Tried to move back new op lower bound from 7022887009803857920 to 7022887007283228672. Current Snapshot: MvccSnapshot[applied={T|T < 7022887009803857920}]
INSERT report
rows total: 2000
time total: 246.311 ms
time per row: 0.123156 ms
I20240501 14:00:24.121254 1998 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:24.132925 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 85842504df3e4571b1a9750809711db2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.133158 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 85842504df3e4571b1a9750809711db2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.133280 2388 tablet_service.cc:1467] Processing CreateTablet for tablet fac8d3842d994279b355f9a18a82f901 (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.133280 2257 tablet_service.cc:1467] Processing CreateTablet for tablet ff4b3345715c44c091ed1073a8a706ff (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.133510 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85842504df3e4571b1a9750809711db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.133541 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85842504df3e4571b1a9750809711db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.133687 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac8d3842d994279b355f9a18a82f901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.133692 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4b3345715c44c091ed1073a8a706ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.135007 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 85842504df3e4571b1a9750809711db2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.135426 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85842504df3e4571b1a9750809711db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.138926 2518 tablet_service.cc:1467] Processing CreateTablet for tablet ff4b3345715c44c091ed1073a8a706ff (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.139343 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4b3345715c44c091ed1073a8a706ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.142925 2516 tablet_service.cc:1467] Processing CreateTablet for tablet fac8d3842d994279b355f9a18a82f901 (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.143308 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac8d3842d994279b355f9a18a82f901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.146541 4408 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:24.147002 2386 tablet_service.cc:1467] Processing CreateTablet for tablet ff4b3345715c44c091ed1073a8a706ff (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.147462 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4b3345715c44c091ed1073a8a706ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.148593 4408 tablet_bootstrap.cc:654] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.150946 2126 tablet_service.cc:1467] Processing CreateTablet for tablet fac8d3842d994279b355f9a18a82f901 (DEFAULT_TABLE table=rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:24.151373 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac8d3842d994279b355f9a18a82f901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:24.152027 4409 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:24.156064 4409 tablet_bootstrap.cc:654] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.157635 4411 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:24.160971 4411 tablet_bootstrap.cc:654] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.161744 4413 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:24.163132 4409 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:24.163415 4409 ts_tablet_manager.cc:1367] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:00:24.163535 4413 tablet_bootstrap.cc:654] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.164398 4409 raft_consensus.cc:348] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.165290 4409 raft_consensus.cc:374] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.165445 4409 raft_consensus.cc:729] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.167212 4409 consensus_queue.cc:260] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.168378 4409 ts_tablet_manager.cc:1398] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:24.168802 4409 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:24.170923 4409 tablet_bootstrap.cc:654] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.173476 4411 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:24.173709 4411 ts_tablet_manager.cc:1367] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240501 14:00:24.174383 4413 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:24.174564 4413 ts_tablet_manager.cc:1367] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240501 14:00:24.174595 4411 raft_consensus.cc:348] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.174819 4411 raft_consensus.cc:374] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.174909 4411 raft_consensus.cc:729] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.175136 4411 consensus_queue.cc:260] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.175385 4413 raft_consensus.cc:348] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.175612 4413 raft_consensus.cc:374] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.175706 4411 ts_tablet_manager.cc:1398] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:24.175724 4408 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:24.175740 4413 raft_consensus.cc:729] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.175869 4408 ts_tablet_manager.cc:1367] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.012s
I20240501 14:00:24.176014 4411 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:24.176122 4413 consensus_queue.cc:260] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.176874 4413 ts_tablet_manager.cc:1398] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:24.177219 4413 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:24.177548 4411 tablet_bootstrap.cc:654] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.178820 4413 tablet_bootstrap.cc:654] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.178797 4408 raft_consensus.cc:348] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.179004 4408 raft_consensus.cc:374] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.179104 4408 raft_consensus.cc:729] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.179271 4409 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:24.179339 4408 consensus_queue.cc:260] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.179469 4409 ts_tablet_manager.cc:1367] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:00:24.179998 4408 ts_tablet_manager.cc:1398] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:24.180279 4408 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:24.180300 4409 raft_consensus.cc:348] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.180536 4409 raft_consensus.cc:374] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.180631 4409 raft_consensus.cc:729] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.180930 4409 consensus_queue.cc:260] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.181458 4409 ts_tablet_manager.cc:1398] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:24.183996 4408 tablet_bootstrap.cc:654] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.186815 4408 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:24.187007 4408 ts_tablet_manager.cc:1367] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240501 14:00:24.187867 4408 raft_consensus.cc:348] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.188174 4408 raft_consensus.cc:374] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.188381 4408 raft_consensus.cc:729] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.188673 4408 consensus_queue.cc:260] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.189889 4408 ts_tablet_manager.cc:1398] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:24.190677 4411 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:24.190861 4411 ts_tablet_manager.cc:1367] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240501 14:00:24.191628 4411 raft_consensus.cc:348] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.191859 4411 raft_consensus.cc:374] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.191938 4411 raft_consensus.cc:729] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.192216 4411 consensus_queue.cc:260] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.192440 4413 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:24.192607 4413 ts_tablet_manager.cc:1367] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240501 14:00:24.192763 4411 ts_tablet_manager.cc:1398] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:24.193050 4411 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:24.193363 4413 raft_consensus.cc:348] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.193563 4413 raft_consensus.cc:374] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.194401 4413 raft_consensus.cc:729] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.194727 4413 consensus_queue.cc:260] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.196478 4411 tablet_bootstrap.cc:654] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:24.196910 4413 ts_tablet_manager.cc:1398] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:24.198253 4411 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:24.198400 4411 ts_tablet_manager.cc:1367] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 14:00:24.199102 4411 raft_consensus.cc:348] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.199355 4411 raft_consensus.cc:374] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:24.199455 4411 raft_consensus.cc:729] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:24.199718 4411 consensus_queue.cc:260] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.200373 4411 ts_tablet_manager.cc:1398] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:24.320645 4421 raft_consensus.cc:482] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:24.320847 4421 raft_consensus.cc:504] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.321492 4421 leader_election.cc:288] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:24.322037 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4b3345715c44c091ed1073a8a706ff" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:24.322100 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4b3345715c44c091ed1073a8a706ff" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:24.322274 2406 raft_consensus.cc:2454] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:24.322300 2277 raft_consensus.cc:2454] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:24.322762 2471 leader_election.cc:302] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:24.323047 4421 raft_consensus.cc:2781] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:24.323172 4421 raft_consensus.cc:482] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:24.323261 4421 raft_consensus.cc:3037] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.325119 4421 raft_consensus.cc:504] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.325655 4421 leader_election.cc:288] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:24.326184 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4b3345715c44c091ed1073a8a706ff" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:24.326195 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4b3345715c44c091ed1073a8a706ff" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:24.326346 2406 raft_consensus.cc:3037] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.326380 2277 raft_consensus.cc:3037] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.328367 2406 raft_consensus.cc:2454] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:24.328375 2277 raft_consensus.cc:2454] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:24.328755 2471 leader_election.cc:302] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:24.329051 4421 raft_consensus.cc:2781] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:24.329291 4421 raft_consensus.cc:686] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:24.329566 4421 consensus_queue.cc:237] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:24.332964 2000 catalog_manager.cc:5484] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:24.482939 4388 raft_consensus.cc:482] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:24.483136 4388 raft_consensus.cc:504] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.483711 4388 leader_election.cc:288] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:24.484090 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac8d3842d994279b355f9a18a82f901" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:24.484167 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac8d3842d994279b355f9a18a82f901" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:24.484292 2538 raft_consensus.cc:2454] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:24.484419 2147 raft_consensus.cc:2454] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:24.484716 2341 leader_election.cc:302] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:24.484953 4388 raft_consensus.cc:2781] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:24.485067 4388 raft_consensus.cc:482] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:24.485144 4388 raft_consensus.cc:3037] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.487093 4388 raft_consensus.cc:504] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.487666 4388 leader_election.cc:288] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:24.488111 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac8d3842d994279b355f9a18a82f901" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:24.488242 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac8d3842d994279b355f9a18a82f901" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:24.488315 2538 raft_consensus.cc:3037] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.488445 2147 raft_consensus.cc:3037] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.490201 2538 raft_consensus.cc:2454] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:24.490316 2147 raft_consensus.cc:2454] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:24.490903 2341 leader_election.cc:302] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:24.491253 4388 raft_consensus.cc:2781] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:24.491446 4388 raft_consensus.cc:686] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:24.491775 4388 consensus_queue.cc:237] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.494880 2000 catalog_manager.cc:5484] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:24.568744 4240 raft_consensus.cc:482] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:24.568931 4240 raft_consensus.cc:504] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.569612 4240 leader_election.cc:288] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:24.570137 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85842504df3e4571b1a9750809711db2" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:24.570183 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85842504df3e4571b1a9750809711db2" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:24.570381 2538 raft_consensus.cc:2454] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:24.570410 2147 raft_consensus.cc:2454] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:24.570850 2211 leader_election.cc:302] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:24.571146 4307 raft_consensus.cc:2781] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:24.571271 4307 raft_consensus.cc:482] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:24.571377 4307 raft_consensus.cc:3037] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.573724 4307 raft_consensus.cc:504] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.574317 4307 leader_election.cc:288] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:24.574808 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85842504df3e4571b1a9750809711db2" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:24.574851 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85842504df3e4571b1a9750809711db2" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:24.575016 2538 raft_consensus.cc:3037] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.575031 2147 raft_consensus.cc:3037] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:24.576975 2147 raft_consensus.cc:2454] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:24.577046 2538 raft_consensus.cc:2454] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:24.577755 2211 leader_election.cc:302] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:24.578311 4307 raft_consensus.cc:2781] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:24.578459 4307 raft_consensus.cc:686] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:24.578786 4307 consensus_queue.cc:237] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:24.582284 2000 catalog_manager.cc:5484] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:24.856429 4421 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.898898 4431 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.959798 4307 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.967293 4307 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:24.980922 4388 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:24.981659 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:24.982098 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:00:24.988458 4392 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:24.993330 4425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
INSERT report
rows total: 2000
time total: 233.869 ms
time per row: 0.116935 ms
I20240501 14:00:25.376296 2004 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:25.385664 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 28f67dedd5f44301b00d3e9c08330198 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.385893 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 28f67dedd5f44301b00d3e9c08330198 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.386086 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f67dedd5f44301b00d3e9c08330198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.386346 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f67dedd5f44301b00d3e9c08330198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.387115 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 28f67dedd5f44301b00d3e9c08330198 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.387388 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef9539679954eb284005573c96789bf (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.387538 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f67dedd5f44301b00d3e9c08330198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.387776 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef9539679954eb284005573c96789bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.390349 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 48b053e242914219aa46965f405b10c4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.390786 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b053e242914219aa46965f405b10c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.390947 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 48b053e242914219aa46965f405b10c4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.391359 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b053e242914219aa46965f405b10c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.394284 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 48b053e242914219aa46965f405b10c4 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.394760 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b053e242914219aa46965f405b10c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.394814 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef9539679954eb284005573c96789bf (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.395223 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef9539679954eb284005573c96789bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.400876 4494 tablet_bootstrap.cc:492] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:25.401794 4493 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:25.402722 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef9539679954eb284005573c96789bf (DEFAULT_TABLE table=rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:25.403189 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef9539679954eb284005573c96789bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:25.403502 4494 tablet_bootstrap.cc:654] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.404335 4493 tablet_bootstrap.cc:654] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.414151 4494 tablet_bootstrap.cc:492] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:25.414538 4494 ts_tablet_manager.cc:1367] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240501 14:00:25.414943 4497 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:25.415513 4494 raft_consensus.cc:348] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.415843 4494 raft_consensus.cc:374] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.415977 4494 raft_consensus.cc:729] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.416379 4494 consensus_queue.cc:260] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.417034 4494 ts_tablet_manager.cc:1398] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:25.417361 4494 tablet_bootstrap.cc:492] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:25.419292 4494 tablet_bootstrap.cc:654] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.421371 4498 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:25.424082 4498 tablet_bootstrap.cc:654] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.424710 4497 tablet_bootstrap.cc:654] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.429693 4493 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:25.429965 4493 ts_tablet_manager.cc:1367] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.015s
I20240501 14:00:25.430212 4497 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:25.430416 4497 ts_tablet_manager.cc:1367] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20240501 14:00:25.430948 4493 raft_consensus.cc:348] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.431365 4493 raft_consensus.cc:374] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.431864 4493 raft_consensus.cc:729] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.432204 4493 consensus_queue.cc:260] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.432960 4493 ts_tablet_manager.cc:1398] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:25.432967 4494 tablet_bootstrap.cc:492] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:25.433202 4494 ts_tablet_manager.cc:1367] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20240501 14:00:25.433344 4493 tablet_bootstrap.cc:492] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:25.433990 4494 raft_consensus.cc:348] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.434218 4494 raft_consensus.cc:374] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.434302 4494 raft_consensus.cc:729] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.434609 4494 consensus_queue.cc:260] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.435304 4493 tablet_bootstrap.cc:654] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.435303 4494 ts_tablet_manager.cc:1398] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:25.440935 4493 tablet_bootstrap.cc:492] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:25.441128 4493 ts_tablet_manager.cc:1367] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 14:00:25.441816 4498 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:25.442044 4493 raft_consensus.cc:348] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.442255 4498 ts_tablet_manager.cc:1367] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20240501 14:00:25.442389 4493 raft_consensus.cc:374] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.442564 4493 raft_consensus.cc:729] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.442894 4493 consensus_queue.cc:260] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.443229 4498 raft_consensus.cc:348] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.443507 4498 raft_consensus.cc:374] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.443650 4498 raft_consensus.cc:729] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.443969 4498 consensus_queue.cc:260] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.444823 4493 ts_tablet_manager.cc:1398] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:00:25.445153 4493 tablet_bootstrap.cc:492] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:25.447032 4493 tablet_bootstrap.cc:654] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.445407 4498 ts_tablet_manager.cc:1398] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:25.449690 4493 tablet_bootstrap.cc:492] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:25.449849 4493 ts_tablet_manager.cc:1367] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:25.450644 4493 raft_consensus.cc:348] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.450834 4493 raft_consensus.cc:374] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.450902 4493 raft_consensus.cc:729] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.451148 4493 consensus_queue.cc:260] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.451776 4498 tablet_bootstrap.cc:492] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:25.451818 4493 ts_tablet_manager.cc:1398] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:25.450492 4497 raft_consensus.cc:348] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.451944 4497 raft_consensus.cc:374] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.452124 4497 raft_consensus.cc:729] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.452684 4497 consensus_queue.cc:260] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.453280 4497 ts_tablet_manager.cc:1398] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:00:25.453769 4497 tablet_bootstrap.cc:492] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:25.455495 4497 tablet_bootstrap.cc:654] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.456740 4498 tablet_bootstrap.cc:654] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:25.464644 4497 tablet_bootstrap.cc:492] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:25.464634 4498 tablet_bootstrap.cc:492] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:25.464826 4498 ts_tablet_manager.cc:1367] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240501 14:00:25.464844 4497 ts_tablet_manager.cc:1367] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20240501 14:00:25.465490 4497 raft_consensus.cc:348] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.465557 4498 raft_consensus.cc:348] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.465688 4497 raft_consensus.cc:374] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.465760 4497 raft_consensus.cc:729] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.465763 4498 raft_consensus.cc:374] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:25.465826 4498 raft_consensus.cc:729] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:25.466008 4497 consensus_queue.cc:260] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.466137 4498 consensus_queue.cc:260] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.466794 4498 ts_tablet_manager.cc:1398] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:25.466796 4497 ts_tablet_manager.cc:1398] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:25.588491 4392 raft_consensus.cc:482] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:25.588716 4392 raft_consensus.cc:504] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.589412 4392 leader_election.cc:288] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:25.590081 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b053e242914219aa46965f405b10c4" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:25.590165 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b053e242914219aa46965f405b10c4" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:25.590451 2146 raft_consensus.cc:2454] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:25.590818 2275 raft_consensus.cc:2454] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:25.590973 2341 leader_election.cc:302] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:25.591284 4392 raft_consensus.cc:2781] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:25.591392 4392 raft_consensus.cc:482] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:25.591499 4392 raft_consensus.cc:3037] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.593295 4392 raft_consensus.cc:504] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.593871 4392 leader_election.cc:288] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:25.594338 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b053e242914219aa46965f405b10c4" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:25.594487 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b053e242914219aa46965f405b10c4" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:25.594580 2275 raft_consensus.cc:3037] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.594735 2146 raft_consensus.cc:3037] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.597297 2275 raft_consensus.cc:2454] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:25.597297 2146 raft_consensus.cc:2454] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:25.597698 2341 leader_election.cc:302] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:25.598023 4392 raft_consensus.cc:2781] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:25.598284 4392 raft_consensus.cc:686] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:25.598587 4392 consensus_queue.cc:237] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:25.602082 2000 catalog_manager.cc:5484] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:25.605847 4341 raft_consensus.cc:482] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:25.606024 4341 raft_consensus.cc:504] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.606842 4341 leader_election.cc:288] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:25.607285 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f67dedd5f44301b00d3e9c08330198" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:25.607452 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f67dedd5f44301b00d3e9c08330198" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:25.607540 2538 raft_consensus.cc:2454] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:25.607712 2275 raft_consensus.cc:2454] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:25.608115 2080 leader_election.cc:302] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:25.608422 4341 raft_consensus.cc:2781] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:25.608541 4341 raft_consensus.cc:482] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:25.608633 4341 raft_consensus.cc:3037] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.610584 4341 raft_consensus.cc:504] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.611192 4341 leader_election.cc:288] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:25.611662 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f67dedd5f44301b00d3e9c08330198" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:25.611702 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f67dedd5f44301b00d3e9c08330198" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:25.611871 2536 raft_consensus.cc:3037] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.611892 2275 raft_consensus.cc:3037] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.613734 2536 raft_consensus.cc:2454] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:25.613787 2275 raft_consensus.cc:2454] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:25.614248 2080 leader_election.cc:302] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:25.614583 4341 raft_consensus.cc:2781] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:25.614743 4341 raft_consensus.cc:686] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:25.615005 4341 consensus_queue.cc:237] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.618069 2000 catalog_manager.cc:5484] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:25.797255 4474 raft_consensus.cc:482] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:25.797437 4474 raft_consensus.cc:504] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.798105 4474 leader_election.cc:288] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:25.798601 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef9539679954eb284005573c96789bf" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:25.798668 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef9539679954eb284005573c96789bf" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:25.798816 2408 raft_consensus.cc:2454] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:25.798887 2536 raft_consensus.cc:2454] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:25.799346 2211 leader_election.cc:302] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:25.799647 4474 raft_consensus.cc:2781] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:25.799806 4474 raft_consensus.cc:482] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:25.799904 4474 raft_consensus.cc:3037] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.812350 4474 raft_consensus.cc:504] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.813133 4474 leader_election.cc:288] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:25.813674 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef9539679954eb284005573c96789bf" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:25.813735 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef9539679954eb284005573c96789bf" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:25.813921 2408 raft_consensus.cc:3037] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.813951 2536 raft_consensus.cc:3037] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:25.816251 2536 raft_consensus.cc:2454] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:25.816728 2211 leader_election.cc:302] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:25.816826 2408 raft_consensus.cc:2454] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:25.817238 4474 raft_consensus.cc:2781] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:25.817400 4474 raft_consensus.cc:686] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:25.817678 4474 consensus_queue.cc:237] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:25.825851 2000 catalog_manager.cc:5484] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:26.033810 4341 consensus_queue.cc:1035] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.041719 4341 consensus_queue.cc:1035] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.046794 4392 consensus_queue.cc:1035] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.068256 4519 consensus_queue.cc:1035] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 14:00:26.240067 4510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:26.240533 4510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:26.248659 4510 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240501 14:00:26.298204 4474 consensus_queue.cc:1035] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:26.305922 4508 consensus_queue.cc:1035] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 248.159 ms
time per row: 0.124079 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:27.549649 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:27.559352 2127 tablet_service.cc:1467] Processing CreateTablet for tablet f67ca7eac95d4a7fbd28667bdde44a22 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.559479 2386 tablet_service.cc:1467] Processing CreateTablet for tablet f67ca7eac95d4a7fbd28667bdde44a22 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.559849 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67ca7eac95d4a7fbd28667bdde44a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.559864 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67ca7eac95d4a7fbd28667bdde44a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.563648 2516 tablet_service.cc:1467] Processing CreateTablet for tablet f67ca7eac95d4a7fbd28667bdde44a22 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.563706 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 98e027e2a36541eda1a67c0a3ab48d65 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.563926 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 98e027e2a36541eda1a67c0a3ab48d65 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.564093 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67ca7eac95d4a7fbd28667bdde44a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.564100 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e027e2a36541eda1a67c0a3ab48d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.564249 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e027e2a36541eda1a67c0a3ab48d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.564239 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 98e027e2a36541eda1a67c0a3ab48d65 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.564642 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e027e2a36541eda1a67c0a3ab48d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.567466 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb2dab6f4684bb18bddc9edba1d6517 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.567873 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2dab6f4684bb18bddc9edba1d6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.563688 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb2dab6f4684bb18bddc9edba1d6517 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.570144 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2dab6f4684bb18bddc9edba1d6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.572450 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 1cb2dab6f4684bb18bddc9edba1d6517 (DEFAULT_TABLE table=rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:27.572881 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2dab6f4684bb18bddc9edba1d6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:27.584252 4580 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:27.587579 4580 tablet_bootstrap.cc:654] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.589497 4581 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:27.589828 4579 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:27.591473 4579 tablet_bootstrap.cc:654] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.591385 4581 tablet_bootstrap.cc:654] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.592993 4582 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:27.593967 4579 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:27.594122 4579 ts_tablet_manager.cc:1367] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:00:27.594939 4582 tablet_bootstrap.cc:654] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.594926 4579 raft_consensus.cc:348] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.595163 4579 raft_consensus.cc:374] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.595263 4579 raft_consensus.cc:729] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.595579 4579 consensus_queue.cc:260] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.595932 4580 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:27.596138 4579 ts_tablet_manager.cc:1398] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:27.596161 4580 ts_tablet_manager.cc:1367] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240501 14:00:27.596437 4579 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:27.597129 4580 raft_consensus.cc:348] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.597426 4580 raft_consensus.cc:374] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.597559 4580 raft_consensus.cc:729] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.597852 4580 consensus_queue.cc:260] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.598021 4579 tablet_bootstrap.cc:654] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.599848 4581 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:27.600044 4581 ts_tablet_manager.cc:1367] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:00:27.600867 4581 raft_consensus.cc:348] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.601101 4581 raft_consensus.cc:374] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.601214 4581 raft_consensus.cc:729] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.601473 4581 consensus_queue.cc:260] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.601837 4580 ts_tablet_manager.cc:1398] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:00:27.602087 4581 ts_tablet_manager.cc:1398] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:27.602998 4581 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:27.604698 4581 tablet_bootstrap.cc:654] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.607383 4579 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:27.607642 4579 ts_tablet_manager.cc:1367] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20240501 14:00:27.608618 4579 raft_consensus.cc:348] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.608909 4579 raft_consensus.cc:374] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.608981 4579 raft_consensus.cc:729] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.609216 4579 consensus_queue.cc:260] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.609282 4582 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:27.609434 4582 ts_tablet_manager.cc:1367] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240501 14:00:27.609768 4579 ts_tablet_manager.cc:1398] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:27.610348 4579 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:27.610486 4582 raft_consensus.cc:348] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.610720 4582 raft_consensus.cc:374] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.610854 4582 raft_consensus.cc:729] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.611132 4582 consensus_queue.cc:260] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.611729 4582 ts_tablet_manager.cc:1398] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:27.611941 4579 tablet_bootstrap.cc:654] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.614915 4582 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:27.617362 4582 tablet_bootstrap.cc:654] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.621145 4579 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:27.621308 4579 ts_tablet_manager.cc:1367] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20240501 14:00:27.622140 4579 raft_consensus.cc:348] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.622387 4579 raft_consensus.cc:374] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.622519 4579 raft_consensus.cc:729] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.622798 4579 consensus_queue.cc:260] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.623708 4579 ts_tablet_manager.cc:1398] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:27.625741 4581 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:27.625926 4581 ts_tablet_manager.cc:1367] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.016s
I20240501 14:00:27.626860 4581 raft_consensus.cc:348] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.627091 4581 raft_consensus.cc:374] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.627242 4581 raft_consensus.cc:729] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.627655 4582 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:27.627560 4581 consensus_queue.cc:260] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.627831 4582 ts_tablet_manager.cc:1367] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 14:00:27.628206 4581 ts_tablet_manager.cc:1398] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:27.628446 4582 raft_consensus.cc:348] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.628629 4582 raft_consensus.cc:374] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.628695 4582 raft_consensus.cc:729] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.628923 4582 consensus_queue.cc:260] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.629549 4582 ts_tablet_manager.cc:1398] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:27.629793 4582 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:27.631665 4582 tablet_bootstrap.cc:654] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:27.634609 4582 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:27.634749 4582 ts_tablet_manager.cc:1367] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240501 14:00:27.635360 4582 raft_consensus.cc:348] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.635521 4582 raft_consensus.cc:374] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:27.635608 4582 raft_consensus.cc:729] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:27.635797 4582 consensus_queue.cc:260] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.636229 4582 ts_tablet_manager.cc:1398] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:27.688902 4431 raft_consensus.cc:482] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:27.689134 4431 raft_consensus.cc:504] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.689911 4431 leader_election.cc:288] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:27.690263 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2dab6f4684bb18bddc9edba1d6517" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:27.690549 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2dab6f4684bb18bddc9edba1d6517" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:27.690601 2146 raft_consensus.cc:2454] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:27.690762 2408 raft_consensus.cc:2454] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:27.691047 2471 leader_election.cc:302] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:27.691329 4431 raft_consensus.cc:2781] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:27.691430 4431 raft_consensus.cc:482] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:27.691526 4431 raft_consensus.cc:3037] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.693511 4431 raft_consensus.cc:504] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.694267 4431 leader_election.cc:288] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:27.695022 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2dab6f4684bb18bddc9edba1d6517" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:27.695021 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cb2dab6f4684bb18bddc9edba1d6517" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:27.695236 2146 raft_consensus.cc:3037] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.695230 2408 raft_consensus.cc:3037] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.698055 2408 raft_consensus.cc:2454] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:27.698055 2146 raft_consensus.cc:2454] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:27.698535 2471 leader_election.cc:302] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:27.698822 4431 raft_consensus.cc:2781] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:27.698997 4431 raft_consensus.cc:686] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:27.699347 4431 consensus_queue.cc:237] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:27.702956 4482 raft_consensus.cc:482] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:27.703110 4482 raft_consensus.cc:504] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.702950 1997 catalog_manager.cc:5484] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:27.703748 4482 leader_election.cc:288] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:27.704213 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67ca7eac95d4a7fbd28667bdde44a22" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:27.704214 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67ca7eac95d4a7fbd28667bdde44a22" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:27.704463 2146 raft_consensus.cc:2454] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:27.704464 2536 raft_consensus.cc:2454] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:27.704859 2341 leader_election.cc:302] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:27.705816 4482 raft_consensus.cc:2781] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:27.705935 4482 raft_consensus.cc:482] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:27.706024 4482 raft_consensus.cc:3037] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.708088 4482 raft_consensus.cc:504] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.708775 4482 leader_election.cc:288] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:27.709179 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67ca7eac95d4a7fbd28667bdde44a22" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:27.709259 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67ca7eac95d4a7fbd28667bdde44a22" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:27.709435 2146 raft_consensus.cc:3037] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.709478 2536 raft_consensus.cc:3037] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.712260 2146 raft_consensus.cc:2454] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:27.712280 2536 raft_consensus.cc:2454] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:27.712771 2341 leader_election.cc:302] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:27.713148 4482 raft_consensus.cc:2781] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:27.713295 4482 raft_consensus.cc:686] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:27.713567 4482 consensus_queue.cc:237] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:27.716792 1997 catalog_manager.cc:5484] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:27.731710 4431 raft_consensus.cc:482] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:27.731899 4431 raft_consensus.cc:504] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.732563 4431 leader_election.cc:288] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:27.733134 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e027e2a36541eda1a67c0a3ab48d65" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:27.733124 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e027e2a36541eda1a67c0a3ab48d65" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:27.733378 2146 raft_consensus.cc:2454] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:27.733379 2275 raft_consensus.cc:2454] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:27.733803 2471 leader_election.cc:302] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:27.734071 4431 raft_consensus.cc:2781] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:27.734190 4431 raft_consensus.cc:482] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:27.734274 4431 raft_consensus.cc:3037] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.736174 4431 raft_consensus.cc:504] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.736750 4431 leader_election.cc:288] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:27.737294 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e027e2a36541eda1a67c0a3ab48d65" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:27.737372 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e027e2a36541eda1a67c0a3ab48d65" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:27.737493 2275 raft_consensus.cc:3037] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.737598 2146 raft_consensus.cc:3037] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:27.739794 2275 raft_consensus.cc:2454] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:27.739933 2146 raft_consensus.cc:2454] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:27.740330 2471 leader_election.cc:302] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:27.740736 4431 raft_consensus.cc:2781] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:27.740912 4431 raft_consensus.cc:686] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:27.741225 4431 consensus_queue.cc:237] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:27.744804 1997 catalog_manager.cc:5484] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:28.098731 4392 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:28.109560 4482 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:28.147259 4596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:28.147601 4596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:28.155761 4596 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:00:28.167635 4431 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:28.173074 4431 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:28.198148 4431 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
I20240501 14:00:28.205547 4574 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 281.724 ms
time per row: 0.140862 ms
I20240501 14:00:28.613615 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:28.624060 2386 tablet_service.cc:1467] Processing CreateTablet for tablet e58200ed1d53430e97ed77e154dee4a7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.624264 2256 tablet_service.cc:1467] Processing CreateTablet for tablet e58200ed1d53430e97ed77e154dee4a7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.624516 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58200ed1d53430e97ed77e154dee4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.624701 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58200ed1d53430e97ed77e154dee4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.630395 2125 tablet_service.cc:1467] Processing CreateTablet for tablet e58200ed1d53430e97ed77e154dee4a7 (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.630878 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58200ed1d53430e97ed77e154dee4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.631211 4656 tablet_bootstrap.cc:492] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:28.632302 4655 tablet_bootstrap.cc:492] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:28.633256 4656 tablet_bootstrap.cc:654] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.634301 4655 tablet_bootstrap.cc:654] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.635581 2257 tablet_service.cc:1467] Processing CreateTablet for tablet a131d188420344e0a76c7b84596194ea (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.636044 2517 tablet_service.cc:1467] Processing CreateTablet for tablet a131d188420344e0a76c7b84596194ea (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.636153 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a131d188420344e0a76c7b84596194ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.636157 2386 tablet_service.cc:1467] Processing CreateTablet for tablet a131d188420344e0a76c7b84596194ea (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.636515 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a131d188420344e0a76c7b84596194ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.636538 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a131d188420344e0a76c7b84596194ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.638903 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 8afb34c628104a51a0d9c9d4dd649123 (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.639310 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afb34c628104a51a0d9c9d4dd649123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.642661 4655 tablet_bootstrap.cc:492] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:28.642865 4655 ts_tablet_manager.cc:1367] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 14:00:28.642935 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 8afb34c628104a51a0d9c9d4dd649123 (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.643168 4656 tablet_bootstrap.cc:492] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:28.643317 4656 ts_tablet_manager.cc:1367] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 14:00:28.643455 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afb34c628104a51a0d9c9d4dd649123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.643858 4655 raft_consensus.cc:348] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:28.644064 4655 raft_consensus.cc:374] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.644165 4655 raft_consensus.cc:729] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.644081 4656 raft_consensus.cc:348] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:28.644271 4656 raft_consensus.cc:374] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.644364 4656 raft_consensus.cc:729] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.644434 4655 consensus_queue.cc:260] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:28.644590 4656 consensus_queue.cc:260] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:28.645156 4656 ts_tablet_manager.cc:1398] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:28.645174 4655 ts_tablet_manager.cc:1398] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:28.646569 4655 tablet_bootstrap.cc:492] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:28.646973 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 8afb34c628104a51a0d9c9d4dd649123 (DEFAULT_TABLE table=rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:28.647401 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afb34c628104a51a0d9c9d4dd649123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:28.648541 4655 tablet_bootstrap.cc:654] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.658474 4655 tablet_bootstrap.cc:492] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:28.658716 4655 ts_tablet_manager.cc:1367] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 14:00:28.659790 4655 raft_consensus.cc:348] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.660032 4655 raft_consensus.cc:374] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.660112 4655 raft_consensus.cc:729] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.660384 4655 consensus_queue.cc:260] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.662264 4655 ts_tablet_manager.cc:1398] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:28.662653 4655 tablet_bootstrap.cc:492] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:28.665953 4655 tablet_bootstrap.cc:654] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.667697 4661 tablet_bootstrap.cc:492] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:28.668143 4595 raft_consensus.cc:482] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:28.668320 4595 raft_consensus.cc:504] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.669061 4595 leader_election.cc:288] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:28.670850 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a131d188420344e0a76c7b84596194ea" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
W20240501 14:00:28.671509 2211 leader_election.cc:341] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:28.671918 4655 tablet_bootstrap.cc:492] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:28.672086 4655 ts_tablet_manager.cc:1367] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
W20240501 14:00:28.672437 2211 leader_election.cc:341] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Not found: Tablet not found: a131d188420344e0a76c7b84596194ea
I20240501 14:00:28.672727 2211 leader_election.cc:302] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:28.672981 4655 raft_consensus.cc:348] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.673158 4595 raft_consensus.cc:2726] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:28.673215 4655 raft_consensus.cc:374] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.673651 4655 raft_consensus.cc:729] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.673923 4655 consensus_queue.cc:260] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.675087 4656 tablet_bootstrap.cc:492] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:28.676926 4656 tablet_bootstrap.cc:654] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.677323 4660 tablet_bootstrap.cc:492] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:28.677868 4655 ts_tablet_manager.cc:1398] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240501 14:00:28.679597 4661 tablet_bootstrap.cc:654] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.679911 4660 tablet_bootstrap.cc:654] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.686890 4660 tablet_bootstrap.cc:492] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:28.687173 4660 ts_tablet_manager.cc:1367] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240501 14:00:28.687556 4661 tablet_bootstrap.cc:492] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:28.687706 4661 ts_tablet_manager.cc:1367] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20240501 14:00:28.688001 4660 raft_consensus.cc:348] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.688184 4660 raft_consensus.cc:374] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.688262 4660 raft_consensus.cc:729] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.688350 4661 raft_consensus.cc:348] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.688539 4661 raft_consensus.cc:374] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.688481 4660 consensus_queue.cc:260] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.688628 4661 raft_consensus.cc:729] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.688860 4661 consensus_queue.cc:260] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.689013 4660 ts_tablet_manager.cc:1398] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:28.689430 4661 ts_tablet_manager.cc:1398] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:28.690160 4656 tablet_bootstrap.cc:492] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:28.690340 4656 ts_tablet_manager.cc:1367] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240501 14:00:28.690419 4661 tablet_bootstrap.cc:492] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:28.691138 4656 raft_consensus.cc:348] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.691418 4656 raft_consensus.cc:374] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.691531 4656 raft_consensus.cc:729] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.691737 4431 raft_consensus.cc:482] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:28.691802 4656 consensus_queue.cc:260] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.691892 4431 raft_consensus.cc:504] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.692462 4656 ts_tablet_manager.cc:1398] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:28.692584 4431 leader_election.cc:288] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:28.692797 4656 tablet_bootstrap.cc:492] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:28.693688 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a131d188420344e0a76c7b84596194ea" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:28.693940 2276 raft_consensus.cc:2454] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:28.694370 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a131d188420344e0a76c7b84596194ea" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:28.694574 4656 tablet_bootstrap.cc:654] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:28.695039 2471 leader_election.cc:341] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:28.695320 2471 leader_election.cc:302] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:28.695572 4431 raft_consensus.cc:2781] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:28.695686 4431 raft_consensus.cc:482] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:28.695765 4431 raft_consensus.cc:3037] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.698525 4431 raft_consensus.cc:504] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.701258 4431 leader_election.cc:288] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:28.701967 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a131d188420344e0a76c7b84596194ea" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:28.701973 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a131d188420344e0a76c7b84596194ea" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:28.702209 2276 raft_consensus.cc:3037] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.702927 4661 tablet_bootstrap.cc:654] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:28.703773 4656 tablet_bootstrap.cc:492] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:28.703970 4656 ts_tablet_manager.cc:1367] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240501 14:00:28.704797 2276 raft_consensus.cc:2454] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:28.704919 4656 raft_consensus.cc:348] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.705189 4656 raft_consensus.cc:374] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.705302 4656 raft_consensus.cc:729] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.705606 4656 consensus_queue.cc:260] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
W20240501 14:00:28.705780 2471 leader_election.cc:341] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:28.706034 2471 leader_election.cc:302] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:28.706465 4431 raft_consensus.cc:2781] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:28.706640 4431 raft_consensus.cc:686] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:28.706897 4431 consensus_queue.cc:237] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:28.707396 4656 ts_tablet_manager.cc:1398] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:28.711249 2000 catalog_manager.cc:5484] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:28.712121 4661 tablet_bootstrap.cc:492] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:28.712301 4661 ts_tablet_manager.cc:1367] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.008s
I20240501 14:00:28.713069 4661 raft_consensus.cc:348] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:28.713294 4661 raft_consensus.cc:374] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:28.713388 4661 raft_consensus.cc:729] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:28.713624 4661 consensus_queue.cc:260] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:28.714212 4661 ts_tablet_manager.cc:1398] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:28.863247 4512 raft_consensus.cc:482] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:28.863440 4512 raft_consensus.cc:504] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.864107 4512 leader_election.cc:288] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:28.864588 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb34c628104a51a0d9c9d4dd649123" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:28.864702 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb34c628104a51a0d9c9d4dd649123" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:28.864960 2407 raft_consensus.cc:2454] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:28.864945 2276 raft_consensus.cc:2454] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:28.865538 2080 leader_election.cc:302] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:28.865932 4512 raft_consensus.cc:2781] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:28.866057 4512 raft_consensus.cc:482] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:28.866137 4512 raft_consensus.cc:3037] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.869038 4512 raft_consensus.cc:504] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.869701 4512 leader_election.cc:288] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:28.870352 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb34c628104a51a0d9c9d4dd649123" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:28.870337 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb34c628104a51a0d9c9d4dd649123" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:28.870602 2275 raft_consensus.cc:3037] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.870608 2407 raft_consensus.cc:3037] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:28.872558 2407 raft_consensus.cc:2454] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:28.872953 2080 leader_election.cc:302] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:28.873279 4512 raft_consensus.cc:2781] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:28.873360 2275 raft_consensus.cc:2454] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:28.873422 4512 raft_consensus.cc:686] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:28.873867 4512 consensus_queue.cc:237] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:28.878371 2000 catalog_manager.cc:5484] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:29.036111 4392 raft_consensus.cc:482] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:29.036293 4392 raft_consensus.cc:504] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:29.036854 4392 leader_election.cc:288] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:29.037367 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a131d188420344e0a76c7b84596194ea" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:29.037425 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a131d188420344e0a76c7b84596194ea" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:29.037635 2275 raft_consensus.cc:2379] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64f34785a7b44fa8070b0452a4e0ddd in current term 1: Already voted for candidate 745ac12076e84effa5bde1ecbd6a32ec in this term.
I20240501 14:00:29.038050 2341 leader_election.cc:302] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:29.038304 4392 raft_consensus.cc:3037] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.039366 4648 raft_consensus.cc:482] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:29.039516 4648 raft_consensus.cc:504] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:29.040025 4648 leader_election.cc:288] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:29.040355 4392 raft_consensus.cc:2726] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:29.040531 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58200ed1d53430e97ed77e154dee4a7" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:29.040613 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58200ed1d53430e97ed77e154dee4a7" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:29.040748 2275 raft_consensus.cc:2454] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:29.040843 2147 raft_consensus.cc:2454] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:29.041148 2341 leader_election.cc:302] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:29.041386 4392 raft_consensus.cc:2781] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:29.041499 4392 raft_consensus.cc:482] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:29.041584 4392 raft_consensus.cc:3037] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.043439 4392 raft_consensus.cc:504] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:29.044009 4392 leader_election.cc:288] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:29.044417 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58200ed1d53430e97ed77e154dee4a7" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:29.044473 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58200ed1d53430e97ed77e154dee4a7" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:29.044631 2275 raft_consensus.cc:3037] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.044683 2147 raft_consensus.cc:3037] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:29.047359 2147 raft_consensus.cc:2454] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:29.047381 2275 raft_consensus.cc:2454] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:29.047824 2341 leader_election.cc:302] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:29.048112 4392 raft_consensus.cc:2781] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:29.048270 4392 raft_consensus.cc:686] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:29.048520 4392 consensus_queue.cc:237] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:29.051743 2000 catalog_manager.cc:5484] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:29.092213 4574 consensus_queue.cc:1035] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.098369 4431 consensus_queue.cc:1035] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.255664 4512 consensus_queue.cc:1035] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.268707 4679 consensus_queue.cc:1035] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.455858 4648 consensus_queue.cc:1035] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:29.469308 4648 consensus_queue.cc:1035] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:29.490839 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:29.491222 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:29.499868 4676 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
INSERT report
rows total: 2000
time total: 244.97 ms
time per row: 0.122485 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:00:30.791193 1998 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:30.802567 2517 tablet_service.cc:1467] Processing CreateTablet for tablet b6c38a8bcb4a4e07aa424fd074d5e17e (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.803068 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c38a8bcb4a4e07aa424fd074d5e17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.807595 2256 tablet_service.cc:1467] Processing CreateTablet for tablet b6c38a8bcb4a4e07aa424fd074d5e17e (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.807600 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 01096bd6f2bf4663b70043f61ecdea5b (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.807780 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 01096bd6f2bf4663b70043f61ecdea5b (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.807786 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 2efec972a06b48b99eb89d454fa214a3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.808022 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01096bd6f2bf4663b70043f61ecdea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.808060 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c38a8bcb4a4e07aa424fd074d5e17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.808148 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efec972a06b48b99eb89d454fa214a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.808439 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 2efec972a06b48b99eb89d454fa214a3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.808818 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efec972a06b48b99eb89d454fa214a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.810642 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 2efec972a06b48b99eb89d454fa214a3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.811014 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efec972a06b48b99eb89d454fa214a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.810907 2125 tablet_service.cc:1467] Processing CreateTablet for tablet b6c38a8bcb4a4e07aa424fd074d5e17e (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.811316 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c38a8bcb4a4e07aa424fd074d5e17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.812294 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01096bd6f2bf4663b70043f61ecdea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.814918 2386 tablet_service.cc:1467] Processing CreateTablet for tablet 01096bd6f2bf4663b70043f61ecdea5b (DEFAULT_TABLE table=rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:30.815356 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01096bd6f2bf4663b70043f61ecdea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:30.826787 4741 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:30.829730 4742 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:30.831198 4741 tablet_bootstrap.cc:654] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.833197 4743 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:30.833756 4742 tablet_bootstrap.cc:654] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.834986 4743 tablet_bootstrap.cc:654] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.837049 4741 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:30.837267 4741 ts_tablet_manager.cc:1367] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:00:30.838124 4741 raft_consensus.cc:348] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:30.838389 4741 raft_consensus.cc:374] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.838533 4741 raft_consensus.cc:729] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.838819 4741 consensus_queue.cc:260] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:30.840623 4741 ts_tablet_manager.cc:1398] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:30.840965 4741 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:30.846225 4741 tablet_bootstrap.cc:654] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.849507 4742 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:30.849681 4740 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:30.849776 4742 ts_tablet_manager.cc:1367] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240501 14:00:30.850905 4742 raft_consensus.cc:348] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:30.851131 4742 raft_consensus.cc:374] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.851274 4742 raft_consensus.cc:729] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.851619 4743 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:30.851511 4742 consensus_queue.cc:260] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:30.851763 4743 ts_tablet_manager.cc:1367] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20240501 14:00:30.852056 4742 ts_tablet_manager.cc:1398] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:30.852717 4742 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:30.852624 4743 raft_consensus.cc:348] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:30.852877 4743 raft_consensus.cc:374] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.853205 4743 raft_consensus.cc:729] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.853557 4743 consensus_queue.cc:260] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:30.853734 4741 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:30.855425 4742 tablet_bootstrap.cc:654] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.855435 4743 ts_tablet_manager.cc:1398] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:30.855662 4741 ts_tablet_manager.cc:1367] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240501 14:00:30.855688 4743 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:30.856706 4741 raft_consensus.cc:348] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:30.857182 4743 tablet_bootstrap.cc:654] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.857889 4741 raft_consensus.cc:374] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.858039 4741 raft_consensus.cc:729] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.858405 4741 consensus_queue.cc:260] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:30.858618 4740 tablet_bootstrap.cc:654] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.860316 4741 ts_tablet_manager.cc:1398] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:30.861886 4742 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:30.862170 4742 ts_tablet_manager.cc:1367] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 14:00:30.862465 4743 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:30.862699 4743 ts_tablet_manager.cc:1367] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240501 14:00:30.863129 4742 raft_consensus.cc:348] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:30.863425 4742 raft_consensus.cc:374] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.863462 4743 raft_consensus.cc:348] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:30.863559 4742 raft_consensus.cc:729] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.863739 4743 raft_consensus.cc:374] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.863857 4743 raft_consensus.cc:729] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.863844 4742 consensus_queue.cc:260] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:30.864157 4743 consensus_queue.cc:260] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:30.864431 4742 ts_tablet_manager.cc:1398] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:30.864768 4742 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:30.864840 4743 ts_tablet_manager.cc:1398] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:30.870903 4742 tablet_bootstrap.cc:654] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.890875 4751 raft_consensus.cc:482] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:30.891218 4751 raft_consensus.cc:504] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:30.893384 4751 leader_election.cc:288] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:30.896252 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01096bd6f2bf4663b70043f61ecdea5b" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:30.897429 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01096bd6f2bf4663b70043f61ecdea5b" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
W20240501 14:00:30.898559 2211 leader_election.cc:341] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:00:30.898919 2211 leader_election.cc:341] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:30.899122 2211 leader_election.cc:302] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:30.899384 4751 raft_consensus.cc:2726] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:30.900753 4740 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:30.900960 4740 ts_tablet_manager.cc:1367] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.051s user 0.020s sys 0.015s
I20240501 14:00:30.901959 4740 raft_consensus.cc:348] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:30.902238 4740 raft_consensus.cc:374] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.902379 4740 raft_consensus.cc:729] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.902596 4742 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:30.902665 4740 consensus_queue.cc:260] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:30.902807 4742 ts_tablet_manager.cc:1367] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.021s
I20240501 14:00:30.903740 4742 raft_consensus.cc:348] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:30.904013 4742 raft_consensus.cc:374] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.904136 4742 raft_consensus.cc:729] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.904465 4742 consensus_queue.cc:260] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:30.905089 4742 ts_tablet_manager.cc:1398] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:30.905542 4740 ts_tablet_manager.cc:1398] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:30.905833 4740 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:30.907871 4740 tablet_bootstrap.cc:654] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:30.909927 4740 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:30.910111 4740 ts_tablet_manager.cc:1367] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:30.910889 4740 raft_consensus.cc:348] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:30.911119 4740 raft_consensus.cc:374] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:30.911206 4740 raft_consensus.cc:729] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:30.911451 4740 consensus_queue.cc:260] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:30.912070 4740 ts_tablet_manager.cc:1398] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:31.074283 4571 raft_consensus.cc:482] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:31.074565 4571 raft_consensus.cc:504] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:31.075210 4571 leader_election.cc:288] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:31.075809 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efec972a06b48b99eb89d454fa214a3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:31.076013 2407 raft_consensus.cc:2454] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:31.076270 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efec972a06b48b99eb89d454fa214a3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:31.076382 2080 leader_election.cc:302] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:31.076476 2536 raft_consensus.cc:2454] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:31.076628 4571 raft_consensus.cc:2781] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:31.076756 4571 raft_consensus.cc:482] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:31.076859 4571 raft_consensus.cc:3037] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.079073 4571 raft_consensus.cc:504] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:31.079656 4571 leader_election.cc:288] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:31.080123 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efec972a06b48b99eb89d454fa214a3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:31.080165 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efec972a06b48b99eb89d454fa214a3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:31.080325 2407 raft_consensus.cc:3037] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.080334 2536 raft_consensus.cc:3037] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.082285 2407 raft_consensus.cc:2454] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:31.083070 2536 raft_consensus.cc:2454] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:31.083236 2080 leader_election.cc:302] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:31.083561 4571 raft_consensus.cc:2781] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:31.083763 4571 raft_consensus.cc:686] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:31.084095 4571 consensus_queue.cc:237] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:31.087527 2000 catalog_manager.cc:5484] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:31.168238 4571 raft_consensus.cc:482] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:31.168449 4571 raft_consensus.cc:504] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:31.169037 4571 leader_election.cc:288] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:31.169621 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c38a8bcb4a4e07aa424fd074d5e17e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:31.169662 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c38a8bcb4a4e07aa424fd074d5e17e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:31.169843 2536 raft_consensus.cc:2454] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:31.169890 2278 raft_consensus.cc:2454] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:31.170269 2080 leader_election.cc:302] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:31.170596 4571 raft_consensus.cc:2781] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:31.170688 4571 raft_consensus.cc:482] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:31.170785 4571 raft_consensus.cc:3037] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.172742 4571 raft_consensus.cc:504] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:31.173365 4571 leader_election.cc:288] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:31.173842 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c38a8bcb4a4e07aa424fd074d5e17e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:31.173913 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c38a8bcb4a4e07aa424fd074d5e17e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:31.174053 2536 raft_consensus.cc:3037] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.174117 2278 raft_consensus.cc:3037] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.176252 2278 raft_consensus.cc:2454] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:31.176252 2536 raft_consensus.cc:2454] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:31.176705 2080 leader_election.cc:302] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:31.177004 4571 raft_consensus.cc:2781] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:31.177207 4571 raft_consensus.cc:686] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:31.177469 4571 consensus_queue.cc:237] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:31.181069 2000 catalog_manager.cc:5484] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:31.207715 4732 raft_consensus.cc:482] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:31.207911 4732 raft_consensus.cc:504] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:31.208631 4732 leader_election.cc:288] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:31.209008 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01096bd6f2bf4663b70043f61ecdea5b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:31.209226 2536 raft_consensus.cc:2454] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:31.209159 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01096bd6f2bf4663b70043f61ecdea5b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:31.209424 2278 raft_consensus.cc:2454] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:31.209599 2341 leader_election.cc:302] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:31.209849 4732 raft_consensus.cc:2781] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:31.209961 4732 raft_consensus.cc:482] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:31.210047 4732 raft_consensus.cc:3037] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.212217 4732 raft_consensus.cc:504] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:31.212841 4732 leader_election.cc:288] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:31.213258 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01096bd6f2bf4663b70043f61ecdea5b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:31.213443 2536 raft_consensus.cc:3037] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.213387 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01096bd6f2bf4663b70043f61ecdea5b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:31.213574 2278 raft_consensus.cc:3037] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:31.215365 2536 raft_consensus.cc:2454] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:31.215550 2278 raft_consensus.cc:2454] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:31.215971 2341 leader_election.cc:302] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:31.216243 4732 raft_consensus.cc:2781] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:31.216483 4732 raft_consensus.cc:686] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:31.216761 4732 consensus_queue.cc:237] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:31.219955 2000 catalog_manager.cc:5484] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:31.520169 4735 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.530622 4571 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.590476 4760 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.599025 4760 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.644838 4682 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:31.651430 4682 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:31.676944 4757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:31.677289 4757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:31.685393 4757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
INSERT report
rows total: 2000
time total: 307.289 ms
time per row: 0.153645 ms
I20240501 14:00:32.151208 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:40059
--local_ip_for_outbound_sockets=127.28.17.5
--tserver_master_addrs=127.28.17.62:33799
--webserver_port=42069
--webserver_interface=127.28.17.5
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:00:32.519433 4817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:32.519783 4817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:32.519924 4817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:32.527909 4817 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:00:32.528041 4817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:32.528201 4817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 14:00:32.538143 4817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:40059
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=42069
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:32.539515 4817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:32.541329 4817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:32.554824 4827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:32.555369 4825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:32.555915 4824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:32.556416 4817 server_base.cc:1034] running on GCE node
I20240501 14:00:32.557032 4817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:32.558032 4817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:32.559252 4817 hybrid_clock.cc:648] HybridClock initialized: now 1714572032559190 us; error 64 us; skew 500 ppm
I20240501 14:00:32.559628 4817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:32.562064 4817 webserver.cc:460] Webserver started at http://127.28.17.5:42069/ using document root <none> and password file <none>
I20240501 14:00:32.563064 4817 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:32.563194 4817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:32.569077 4817 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:00:32.572659 4833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:32.574219 4817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:32.574421 4817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 14:00:32.575450 4817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:32.593746 4817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:32.594731 4817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:32.595072 4817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:32.596283 4817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:32.597999 4817 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:32.598111 4817 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:32.598245 4817 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:32.598310 4817 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:32.629902 4817 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:40059
I20240501 14:00:32.629935 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:40059 every 8 connection(s)
I20240501 14:00:32.631461 4817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:00:32.633438 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 4817
I20240501 14:00:32.634048 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.6:46037
--local_ip_for_outbound_sockets=127.28.17.6
--tserver_master_addrs=127.28.17.62:33799
--webserver_port=37937
--webserver_interface=127.28.17.6
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:00:32.646780 4946 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 14:00:32.647331 4946 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:32.648478 4946 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 14:00:32.650117 2000 ts_manager.cc:194] Re-registered known tserver with Master: f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:32.650997 2000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:40711
W20240501 14:00:32.992499 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:32.992812 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:32.992935 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:33.000715 4950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:00:33.000888 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:33.001060 4950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.6
I20240501 14:00:33.011996 4950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.6:46037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.17.6
--webserver_port=37937
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.6
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:33.013289 4950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:33.015009 4950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:33.027184 4957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.027729 4959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.028537 4956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:33.028966 4950 server_base.cc:1034] running on GCE node
I20240501 14:00:33.030345 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:33.031525 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:33.032743 4950 hybrid_clock.cc:648] HybridClock initialized: now 1714572033032670 us; error 74 us; skew 500 ppm
I20240501 14:00:33.033152 4950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:33.035584 4950 webserver.cc:460] Webserver started at http://127.28.17.6:37937/ using document root <none> and password file <none>
I20240501 14:00:33.036326 4950 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:33.036440 4950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:33.042131 4950 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:00:33.045608 4966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.047333 4950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:33.047525 4950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
format_stamp: "Formatted at 2024-05-01 13:59:58 on dist-test-slave-dx8c"
I20240501 14:00:33.048362 4950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:33.070947 4950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:33.071822 4950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:33.072088 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:33.073060 4950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:33.074694 4950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:33.074769 4950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.074887 4950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:33.074932 4950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.115437 4950 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.6:46037
I20240501 14:00:33.115478 5078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.6:46037 every 8 connection(s)
I20240501 14:00:33.116937 4950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:00:33.125353 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 4950
I20240501 14:00:33.125973 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.7:41601
--local_ip_for_outbound_sockets=127.28.17.7
--tserver_master_addrs=127.28.17.62:33799
--webserver_port=37329
--webserver_interface=127.28.17.7
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:00:33.129910 5079 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 14:00:33.130242 5079 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:33.131273 5079 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 14:00:33.133023 2000 ts_manager.cc:194] Re-registered known tserver with Master: 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:33.134107 2000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.6:34793
W20240501 14:00:33.495920 5082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:33.496239 5082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:33.496378 5082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:33.504386 5082 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:00:33.504532 5082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:33.504683 5082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.7
I20240501 14:00:33.515501 5082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.7:41601
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.17.7
--webserver_port=37329
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.7
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:33.516945 5082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:33.518769 5082 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:33.532549 5087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.533325 5090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.533373 5088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:33.536160 5082 server_base.cc:1034] running on GCE node
I20240501 14:00:33.536790 5082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:33.537829 5082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:33.539284 5082 hybrid_clock.cc:648] HybridClock initialized: now 1714572033539220 us; error 41 us; skew 500 ppm
I20240501 14:00:33.539716 5082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:33.542279 5082 webserver.cc:460] Webserver started at http://127.28.17.7:37329/ using document root <none> and password file <none>
I20240501 14:00:33.543129 5082 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:33.543251 5082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:33.550602 5082 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:00:33.553779 5096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.555321 5082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:33.555485 5082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 14:00:33.556301 5082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:33.570602 5082 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:33.571482 5082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:33.571749 5082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:33.572820 5082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:33.574831 5082 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:33.574918 5082 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.575034 5082 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:33.575083 5082 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.621594 5082 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.7:41601
I20240501 14:00:33.621635 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.7:41601 every 8 connection(s)
I20240501 14:00:33.623245 5082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:00:33.628916 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 5082
I20240501 14:00:33.629621 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.8:34221
--local_ip_for_outbound_sockets=127.28.17.8
--tserver_master_addrs=127.28.17.62:33799
--webserver_port=38337
--webserver_interface=127.28.17.8
--builtin_ntp_servers=127.28.17.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:00:33.639308 5209 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 14:00:33.639669 5209 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:33.640695 5209 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 14:00:33.642355 2000 ts_manager.cc:194] Re-registered known tserver with Master: c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:33.643344 2000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.7:54817
I20240501 14:00:33.653581 4946 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
W20240501 14:00:34.010133 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:00:34.010473 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:00:34.010639 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:00:34.019766 5212 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:00:34.019923 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:00:34.020105 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.8
I20240501 14:00:34.030972 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.8:34221
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.17.8
--webserver_port=38337
--tserver_master_addrs=127.28.17.62:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.8
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:00:34.032526 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:00:34.034451 5212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:34.047305 5219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:34.047636 5218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:34.048230 5221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:34.048779 5212 server_base.cc:1034] running on GCE node
I20240501 14:00:34.049423 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:00:34.050547 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:00:34.051788 5212 hybrid_clock.cc:648] HybridClock initialized: now 1714572034051717 us; error 57 us; skew 500 ppm
I20240501 14:00:34.052103 5212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:34.055160 5212 webserver.cc:460] Webserver started at http://127.28.17.8:38337/ using document root <none> and password file <none>
I20240501 14:00:34.055931 5212 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:34.056061 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:34.061969 5212 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20240501 14:00:34.065004 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:34.066529 5212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:34.066706 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
format_stamp: "Formatted at 2024-05-01 13:59:59 on dist-test-slave-dx8c"
I20240501 14:00:34.067607 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:34.108435 5212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:34.109481 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:00:34.109828 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:34.111057 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:34.112854 5212 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:34.112941 5212 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:34.113063 5212 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:34.113127 5212 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:34.136785 5079 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
I20240501 14:00:34.150931 5212 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.8:34221
I20240501 14:00:34.150985 5340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.8:34221 every 8 connection(s)
I20240501 14:00:34.152437 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:00:34.153331 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 5212
I20240501 14:00:34.164716 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:34.169018 5341 heartbeater.cc:347] Connected to a master server at 127.28.17.62:33799
I20240501 14:00:34.169363 5341 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:34.170354 5341 heartbeater.cc:511] Master 127.28.17.62:33799 requested a full tablet report, sending...
I20240501 14:00:34.172168 1999 ts_manager.cc:194] Re-registered known tserver with Master: 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:34.173525 1999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.8:42345
I20240501 14:00:34.192164 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8533119c214142a761d079ca4ee30a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]), partition=
I20240501 14:00:34.192612 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8533119c214142a761d079ca4ee30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.233951 5358 tablet_bootstrap.cc:492] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:34.235298 5358 tablet_bootstrap.cc:654] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.236938 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8533119c214142a761d079ca4ee30a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]), partition=
I20240501 14:00:34.238101 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8533119c214142a761d079ca4ee30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.242981 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8533119c214142a761d079ca4ee30a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]), partition=
I20240501 14:00:34.244292 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8533119c214142a761d079ca4ee30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.254029 5358 tablet_bootstrap.cc:492] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:34.254307 5358 ts_tablet_manager.cc:1367] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.014s
I20240501 14:00:34.255196 5358 raft_consensus.cc:348] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.255604 5358 raft_consensus.cc:374] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.255746 5358 raft_consensus.cc:729] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.256052 5358 consensus_queue.cc:260] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.256647 5358 ts_tablet_manager.cc:1398] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:34.263036 5360 tablet_bootstrap.cc:492] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:34.266285 5360 tablet_bootstrap.cc:654] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.266486 5361 tablet_bootstrap.cc:492] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:34.268522 5360 log.cc:824] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:34.270776 5361 tablet_bootstrap.cc:654] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.272091 5361 log.cc:824] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:34.274117 5360 tablet_bootstrap.cc:492] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:34.274489 5360 ts_tablet_manager.cc:1367] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20240501 14:00:34.288012 5360 raft_consensus.cc:348] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.288422 5360 raft_consensus.cc:374] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.288547 5360 raft_consensus.cc:729] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.289183 5360 consensus_queue.cc:260] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.289860 5361 tablet_bootstrap.cc:492] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:34.290200 5361 ts_tablet_manager.cc:1367] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.014s
I20240501 14:00:34.290855 5360 ts_tablet_manager.cc:1398] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.016s user 0.007s sys 0.005s
I20240501 14:00:34.298434 5361 raft_consensus.cc:348] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.298856 5361 raft_consensus.cc:374] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.298940 5361 raft_consensus.cc:729] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.299607 5361 consensus_queue.cc:260] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.300848 5341 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
I20240501 14:00:34.301340 5361 ts_tablet_manager.cc:1398] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20240501 14:00:34.302021 2324 tablet.cc:2369] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:34.326032 5364 raft_consensus.cc:482] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:34.326397 5364 raft_consensus.cc:504] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.329931 5364 leader_election.cc:288] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:34.337855 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8533119c214142a761d079ca4ee30a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:34.338452 5296 raft_consensus.cc:2454] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:34.339836 4836 leader_election.cc:302] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:34.340492 5364 raft_consensus.cc:2781] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:34.340627 5364 raft_consensus.cc:482] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:34.340728 5364 raft_consensus.cc:3037] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.340852 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8533119c214142a761d079ca4ee30a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:34.341085 2278 raft_consensus.cc:2454] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:34.343343 5364 raft_consensus.cc:504] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.344493 5364 leader_election.cc:288] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:34.344543 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8533119c214142a761d079ca4ee30a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:34.344622 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8533119c214142a761d079ca4ee30a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:34.344756 2278 raft_consensus.cc:3037] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.344837 5296 raft_consensus.cc:3037] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.347329 2278 raft_consensus.cc:2454] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:34.348726 4834 leader_election.cc:302] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:34.349105 5364 raft_consensus.cc:2781] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:34.349972 5296 raft_consensus.cc:2454] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:34.364485 5364 raft_consensus.cc:686] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:34.365101 5364 consensus_queue.cc:237] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:34.370325 1999 catalog_manager.cc:5484] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
W20240501 14:00:34.384406 4947 tablet.cc:2369] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:00:34.404578 5342 tablet.cc:2369] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:34.489447 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:34.498847 2278 raft_consensus.cc:1261] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:34.499697 5296 raft_consensus.cc:1261] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:34.501008 5364 consensus_queue.cc:1035] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240501 14:00:34.506884 5369 consensus_queue.cc:1035] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:34.526861 2000 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]). Expected version 1 got 0
I20240501 14:00:34.528823 1999 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]). Expected version 1 got 0
I20240501 14:00:34.533514 1999 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]). Expected version 1 got 0
I20240501 14:00:34.535879 5377 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:34.537220 5378 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:34.537601 5377 tablet.cc:1620] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:34.538477 5378 tablet.cc:1620] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:34.540799 1999 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]). Expected version 1 got 0
I20240501 14:00:34.553500 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8] alter complete (version 1)
I20240501 14:00:34.555105 5379 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:34.556542 5379 tablet.cc:1620] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:34.557147 1999 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8]). Expected version 1 got 0
I20240501 14:00:34.646946 5209 heartbeater.cc:503] Master 127.28.17.62:33799 was elected leader, sending a full tablet report...
I20240501 14:00:34.656718 1999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:00:34.656926 1999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:00:34.660321 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:34.661032 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 65f0041cd67f4f9baa2b4bf56b1cc2be
I20240501 14:00:34.661725 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 096a3e3778e9449ba5a12bbc91ef6486
I20240501 14:00:34.662065 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:34.662309 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:34.662526 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f0041cd67f4f9baa2b4bf56b1cc2be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:34.662926 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet b3c4f120e9884911b9f6f5158f725bc2
I20240501 14:00:34.663286 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 096a3e3778e9449ba5a12bbc91ef6486 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:34.663556 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 096a3e3778e9449ba5a12bbc91ef6486 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:34.663838 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c4f120e9884911b9f6f5158f725bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:34.664482 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 096a3e3778e9449ba5a12bbc91ef6486 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:34.670322 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c4f120e9884911b9f6f5158f725bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:34.671521 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c4f120e9884911b9f6f5158f725bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:34.673131 5377 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:34.673718 5377 tablet.cc:1620] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:34.676386 5378 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:34.676870 5378 tablet.cc:1620] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:34.677809 5379 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:34.678334 5379 tablet.cc:1620] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:34.680785 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:34.690097 5390 tablet_replica.cc:330] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:34.690752 5390 raft_consensus.cc:2229] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:34.691504 5390 raft_consensus.cc:2258] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.696465 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=16a02ec9273048cba1eda9b4bedb30b8] alter complete (version 2)
I20240501 14:00:34.700029 5276 tablet_service.cc:1467] Processing CreateTablet for tablet fb57ae3a353f4f9e94a47a72ad8792b4 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.700676 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb57ae3a353f4f9e94a47a72ad8792b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.700749 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 052964af500c427e836f0f3121aa91f3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.701246 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052964af500c427e836f0f3121aa91f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.703572 4881 tablet_service.cc:1467] Processing CreateTablet for tablet aaaae937ea4d44068d3ad5b9024b9421 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.704191 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaae937ea4d44068d3ad5b9024b9421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.704739 5389 tablet_replica.cc:330] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:34.705224 5389 raft_consensus.cc:2229] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:34.705582 5389 raft_consensus.cc:2258] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.706993 5389 ts_tablet_manager.cc:1875] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.708479 2127 tablet_service.cc:1467] Processing CreateTablet for tablet aaaae937ea4d44068d3ad5b9024b9421 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.708885 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaae937ea4d44068d3ad5b9024b9421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.715461 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 052964af500c427e836f0f3121aa91f3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.716027 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052964af500c427e836f0f3121aa91f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.722674 5361 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:34.724490 5361 tablet_bootstrap.cc:654] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.728029 5397 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:34.729164 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 052964af500c427e836f0f3121aa91f3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.730665 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052964af500c427e836f0f3121aa91f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.731340 5398 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:34.728466 5143 tablet_service.cc:1467] Processing CreateTablet for tablet fb57ae3a353f4f9e94a47a72ad8792b4 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.735298 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb57ae3a353f4f9e94a47a72ad8792b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.736203 5389 ts_tablet_manager.cc:1888] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.736387 5389 log.cc:1196] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/65f0041cd67f4f9baa2b4bf56b1cc2be
I20240501 14:00:34.736785 5360 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:34.738662 5360 tablet_bootstrap.cc:654] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.736263 5390 ts_tablet_manager.cc:1875] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.738873 5144 tablet_service.cc:1467] Processing CreateTablet for tablet aaaae937ea4d44068d3ad5b9024b9421 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.739326 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaae937ea4d44068d3ad5b9024b9421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.740608 5360 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:34.740767 5360 ts_tablet_manager.cc:1367] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:34.741786 5360 raft_consensus.cc:348] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:34.742089 5360 raft_consensus.cc:374] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.742259 5360 raft_consensus.cc:729] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.742964 5360 consensus_queue.cc:260] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:34.745740 5014 tablet_service.cc:1467] Processing CreateTablet for tablet fb57ae3a353f4f9e94a47a72ad8792b4 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.746404 5360 ts_tablet_manager.cc:1398] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:00:34.747709 5391 tablet_replica.cc:330] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:34.749848 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb57ae3a353f4f9e94a47a72ad8792b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.751528 5398 tablet_bootstrap.cc:654] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.748194 5391 raft_consensus.cc:2229] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:34.750006 5397 tablet_bootstrap.cc:654] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.753239 5391 raft_consensus.cc:2258] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.757385 5390 ts_tablet_manager.cc:1888] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.759335 5389 ts_tablet_manager.cc:1909] T 65f0041cd67f4f9baa2b4bf56b1cc2be P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:34.763022 5389 tablet_replica.cc:330] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:34.763305 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 65f0041cd67f4f9baa2b4bf56b1cc2be (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.765476 5391 ts_tablet_manager.cc:1875] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.767120 5397 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:34.767524 5397 ts_tablet_manager.cc:1367] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.000s
I20240501 14:00:34.768522 5390 log.cc:1196] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/65f0041cd67f4f9baa2b4bf56b1cc2be
I20240501 14:00:34.768548 5397 raft_consensus.cc:348] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:34.769016 5389 raft_consensus.cc:2229] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:34.769611 5389 raft_consensus.cc:2258] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.769696 5390 ts_tablet_manager.cc:1909] T 65f0041cd67f4f9baa2b4bf56b1cc2be P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:34.769780 5397 raft_consensus.cc:374] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.770619 5389 ts_tablet_manager.cc:1875] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.770795 5397 raft_consensus.cc:729] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.771144 5397 consensus_queue.cc:260] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:34.771788 5397 ts_tablet_manager.cc:1398] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:34.779090 5389 ts_tablet_manager.cc:1888] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.779497 5392 tablet_replica.cc:330] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:34.780390 5392 raft_consensus.cc:2229] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:34.780823 5392 raft_consensus.cc:2258] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.782851 5390 tablet_replica.cc:330] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:34.782999 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 65f0041cd67f4f9baa2b4bf56b1cc2be (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.783237 5390 raft_consensus.cc:2229] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:34.783612 5392 ts_tablet_manager.cc:1875] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.783792 5390 raft_consensus.cc:2258] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.784035 5389 log.cc:1196] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/096a3e3778e9449ba5a12bbc91ef6486
I20240501 14:00:34.784689 5390 ts_tablet_manager.cc:1875] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.784746 5361 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:34.784829 5389 ts_tablet_manager.cc:1909] T 096a3e3778e9449ba5a12bbc91ef6486 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:34.784961 5361 ts_tablet_manager.cc:1367] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.005s
I20240501 14:00:34.786077 5361 raft_consensus.cc:348] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:34.786315 5361 raft_consensus.cc:374] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.786438 5361 raft_consensus.cc:729] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.786440 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 096a3e3778e9449ba5a12bbc91ef6486 (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.786770 5361 consensus_queue.cc:260] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:34.787410 5361 ts_tablet_manager.cc:1398] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:34.787735 5361 tablet_bootstrap.cc:492] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:34.789532 5361 tablet_bootstrap.cc:654] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.792407 5391 ts_tablet_manager.cc:1888] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.792721 5391 log.cc:1196] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/65f0041cd67f4f9baa2b4bf56b1cc2be
I20240501 14:00:34.793991 5391 ts_tablet_manager.cc:1909] T 65f0041cd67f4f9baa2b4bf56b1cc2be P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:34.795295 5404 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:34.798472 5391 tablet_replica.cc:330] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:34.798472 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 65f0041cd67f4f9baa2b4bf56b1cc2be (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.798808 5391 raft_consensus.cc:2229] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:34.798862 5398 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:34.799085 5398 ts_tablet_manager.cc:1367] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.013s
I20240501 14:00:34.799235 5391 raft_consensus.cc:2258] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.800005 5391 ts_tablet_manager.cc:1875] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.799932 5398 raft_consensus.cc:348] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:34.800192 5398 raft_consensus.cc:374] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.800283 5398 raft_consensus.cc:729] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.800470 5390 ts_tablet_manager.cc:1888] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.800558 5398 consensus_queue.cc:260] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:34.800698 5390 log.cc:1196] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/b3c4f120e9884911b9f6f5158f725bc2
I20240501 14:00:34.801244 5398 ts_tablet_manager.cc:1398] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:34.801451 5390 ts_tablet_manager.cc:1909] T b3c4f120e9884911b9f6f5158f725bc2 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:34.809700 5391 ts_tablet_manager.cc:1888] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.809931 5391 log.cc:1196] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/096a3e3778e9449ba5a12bbc91ef6486
I20240501 14:00:34.810400 5391 ts_tablet_manager.cc:1909] T 096a3e3778e9449ba5a12bbc91ef6486 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:34.811470 5391 tablet_replica.cc:330] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:34.811728 5391 raft_consensus.cc:2229] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:34.812013 5391 raft_consensus.cc:2258] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.812759 5391 ts_tablet_manager.cc:1875] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.814484 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet b3c4f120e9884911b9f6f5158f725bc2 (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.815901 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 096a3e3778e9449ba5a12bbc91ef6486 (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.820632 5391 ts_tablet_manager.cc:1888] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.820735 5392 ts_tablet_manager.cc:1888] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.820829 5391 log.cc:1196] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/b3c4f120e9884911b9f6f5158f725bc2
I20240501 14:00:34.821408 5361 tablet_bootstrap.cc:492] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:34.821465 5391 ts_tablet_manager.cc:1909] T b3c4f120e9884911b9f6f5158f725bc2 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:34.821669 5361 ts_tablet_manager.cc:1367] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.005s
I20240501 14:00:34.821604 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:34.822567 5361 raft_consensus.cc:348] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.822988 5361 raft_consensus.cc:374] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.823096 5361 raft_consensus.cc:729] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.823408 5361 consensus_queue.cc:260] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.825320 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet b3c4f120e9884911b9f6f5158f725bc2 (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.825883 5392 log.cc:1196] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/096a3e3778e9449ba5a12bbc91ef6486
I20240501 14:00:34.826524 5392 ts_tablet_manager.cc:1909] T 096a3e3778e9449ba5a12bbc91ef6486 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:34.826536 5361 ts_tablet_manager.cc:1398] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:34.831218 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 096a3e3778e9449ba5a12bbc91ef6486 (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.833755 5392 tablet_replica.cc:330] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:34.834219 5404 tablet_bootstrap.cc:654] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.835367 5377 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:34.839121 5378 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:34.839697 5404 log.cc:824] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:34.840360 5403 tablet_bootstrap.cc:492] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:34.841902 5392 raft_consensus.cc:2229] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:34.842697 5392 raft_consensus.cc:2258] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.843979 5392 ts_tablet_manager.cc:1875] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.845341 5379 tablet.cc:1722] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:34.845485 5403 tablet_bootstrap.cc:654] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.847921 5403 log.cc:824] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Log is configured to *not* fsync() on all Append() calls
I20240501 14:00:34.856701 5392 ts_tablet_manager.cc:1888] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:34.856923 5392 log.cc:1196] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/b3c4f120e9884911b9f6f5158f725bc2
I20240501 14:00:34.858846 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=16a02ec9273048cba1eda9b4bedb30b8] alter complete (version 3)
I20240501 14:00:34.859074 5392 ts_tablet_manager.cc:1909] T b3c4f120e9884911b9f6f5158f725bc2 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:34.861940 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet b3c4f120e9884911b9f6f5158f725bc2 (table rebalance_test_table_0 [id=14d46cb81dc945a5845cebc7742282ce]) successfully deleted
I20240501 14:00:34.862732 5403 tablet_bootstrap.cc:492] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:34.863000 5403 ts_tablet_manager.cc:1367] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
W20240501 14:00:34.863848 2193 tablet.cc:2369] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:34.867489 5404 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:34.867834 5404 ts_tablet_manager.cc:1367] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.000s
I20240501 14:00:34.876662 5404 raft_consensus.cc:348] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:34.877195 5404 raft_consensus.cc:374] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.877310 5404 raft_consensus.cc:729] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.878232 5404 consensus_queue.cc:260] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:34.885268 5404 ts_tablet_manager.cc:1398] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20240501 14:00:34.887655 5403 raft_consensus.cc:348] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.888137 5403 raft_consensus.cc:374] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.888238 5403 raft_consensus.cc:729] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.889117 5403 consensus_queue.cc:260] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.891896 5403 ts_tablet_manager.cc:1398] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.029s user 0.012s sys 0.000s
I20240501 14:00:34.897152 5404 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:34.898607 5404 tablet_bootstrap.cc:654] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.904435 5404 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:34.904596 5404 ts_tablet_manager.cc:1367] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:00:34.905441 5404 raft_consensus.cc:348] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:34.905627 5404 raft_consensus.cc:374] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.905725 5404 raft_consensus.cc:729] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.905937 5404 consensus_queue.cc:260] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:34.906553 5404 ts_tablet_manager.cc:1398] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:34.906865 5404 tablet_bootstrap.cc:492] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:34.908983 5404 tablet_bootstrap.cc:654] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.921169 5404 tablet_bootstrap.cc:492] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:34.921361 5404 ts_tablet_manager.cc:1367] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240501 14:00:34.922391 5404 raft_consensus.cc:348] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.923161 5404 raft_consensus.cc:374] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.923529 5404 raft_consensus.cc:729] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.923820 5404 consensus_queue.cc:260] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.924419 5404 ts_tablet_manager.cc:1398] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:34.935159 5408 raft_consensus.cc:482] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:34.935473 5408 raft_consensus.cc:504] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.937232 5408 leader_election.cc:288] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:34.949113 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57ae3a353f4f9e94a47a72ad8792b4" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:34.949745 5034 raft_consensus.cc:2454] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:34.950626 5099 leader_election.cc:302] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:34.951114 5408 raft_consensus.cc:2781] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:34.951274 5408 raft_consensus.cc:482] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:34.951375 5408 raft_consensus.cc:3037] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.952849 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57ae3a353f4f9e94a47a72ad8792b4" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:34.953109 5296 raft_consensus.cc:2454] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:34.954550 5408 raft_consensus.cc:504] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.955413 5408 leader_election.cc:288] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:34.956732 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57ae3a353f4f9e94a47a72ad8792b4" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:34.956847 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb57ae3a353f4f9e94a47a72ad8792b4" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:34.956947 5034 raft_consensus.cc:3037] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.957068 5296 raft_consensus.cc:3037] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.960863 5034 raft_consensus.cc:2454] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:34.961544 5099 leader_election.cc:302] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:34.962002 5408 raft_consensus.cc:2781] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:34.962118 5296 raft_consensus.cc:2454] T fb57ae3a353f4f9e94a47a72ad8792b4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:34.964208 2002 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:34.964517 2002 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:34.966174 5408 raft_consensus.cc:686] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:34.969596 2002 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 3a8533119c214142a761d079ca4ee30a
I20240501 14:00:34.970901 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a8533119c214142a761d079ca4ee30a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:34.971276 5389 tablet_replica.cc:330] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:34.971280 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a8533119c214142a761d079ca4ee30a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:34.971518 5389 raft_consensus.cc:2229] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:34.971755 5389 raft_consensus.cc:2258] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.971639 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a8533119c214142a761d079ca4ee30a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:34 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:34.973032 5389 ts_tablet_manager.cc:1875] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:34.980379 5389 ts_tablet_manager.cc:1888] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:34.981012 5408 consensus_queue.cc:237] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:34.982461 5389 log.cc:1196] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a8533119c214142a761d079ca4ee30a
I20240501 14:00:34.983043 5389 ts_tablet_manager.cc:1909] T 3a8533119c214142a761d079ca4ee30a P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:34.984730 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table_new_name [id=16a02ec9273048cba1eda9b4bedb30b8]) successfully deleted
I20240501 14:00:34.992185 5429 tablet_replica.cc:330] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:34.995734 5429 raft_consensus.cc:2229] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:34.996533 5429 raft_consensus.cc:2258] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:34.998740 5429 ts_tablet_manager.cc:1875] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:00:35.004333 2584 tablet.cc:2369] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:35.006924 5429 ts_tablet_manager.cc:1888] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:35.007860 5428 tablet_replica.cc:330] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
W20240501 14:00:35.008105 2471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:41309, user_credentials={real_user=slave}} blocked reactor thread for 45362.6us
I20240501 14:00:35.008277 2003 catalog_manager.cc:5484] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:35.007323 5429 log.cc:1196] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/3a8533119c214142a761d079ca4ee30a
I20240501 14:00:35.012112 5429 ts_tablet_manager.cc:1909] T 3a8533119c214142a761d079ca4ee30a P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:35.014678 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table_new_name [id=16a02ec9273048cba1eda9b4bedb30b8]) successfully deleted
I20240501 14:00:35.015501 5428 raft_consensus.cc:2229] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.016813 5428 raft_consensus.cc:2258] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.021149 5428 ts_tablet_manager.cc:1875] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.031896 5428 ts_tablet_manager.cc:1888] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:35.032120 5428 log.cc:1196] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/3a8533119c214142a761d079ca4ee30a
I20240501 14:00:35.032680 5428 ts_tablet_manager.cc:1909] T 3a8533119c214142a761d079ca4ee30a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:35.034232 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 3a8533119c214142a761d079ca4ee30a (table rebalancer_dynamic_table_new_name [id=16a02ec9273048cba1eda9b4bedb30b8]) successfully deleted
I20240501 14:00:35.037554 5364 raft_consensus.cc:482] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:35.037735 5364 raft_consensus.cc:504] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.038585 5364 leader_election.cc:288] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:35.074470 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:35.085567 2388 tablet_service.cc:1467] Processing CreateTablet for tablet aa55eb93e93c4c01a4353d20d7fac6ea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602]), partition=
I20240501 14:00:35.085672 4881 tablet_service.cc:1467] Processing CreateTablet for tablet aa55eb93e93c4c01a4353d20d7fac6ea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602]), partition=
I20240501 14:00:35.086025 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa55eb93e93c4c01a4353d20d7fac6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.086089 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa55eb93e93c4c01a4353d20d7fac6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.087185 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaae937ea4d44068d3ad5b9024b9421" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:35.087597 5164 raft_consensus.cc:2454] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:35.088167 4834 leader_election.cc:302] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:35.088493 5364 raft_consensus.cc:2781] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:35.088589 5364 raft_consensus.cc:482] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:35.088662 5364 raft_consensus.cc:3037] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.089196 5014 tablet_service.cc:1467] Processing CreateTablet for tablet aa55eb93e93c4c01a4353d20d7fac6ea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602]), partition=
I20240501 14:00:35.089555 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa55eb93e93c4c01a4353d20d7fac6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.091224 5364 raft_consensus.cc:504] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.091869 5364 leader_election.cc:288] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:35.093183 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaae937ea4d44068d3ad5b9024b9421" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:35.093398 5164 raft_consensus.cc:3037] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.098152 5164 raft_consensus.cc:2454] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:35.099773 4834 leader_election.cc:302] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:35.100775 5360 tablet_bootstrap.cc:492] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:35.100943 5364 raft_consensus.cc:2781] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:35.101107 5364 raft_consensus.cc:686] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:35.101380 5364 consensus_queue.cc:237] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.102458 5360 tablet_bootstrap.cc:654] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.104336 5403 tablet_bootstrap.cc:492] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:35.105892 5403 tablet_bootstrap.cc:654] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.108470 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaae937ea4d44068d3ad5b9024b9421" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:35.108748 2147 raft_consensus.cc:2454] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:35.109172 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaae937ea4d44068d3ad5b9024b9421" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:35.109364 2145 raft_consensus.cc:3037] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.112418 2145 raft_consensus.cc:2454] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
W20240501 14:00:35.119882 5080 tablet.cc:2369] T fb57ae3a353f4f9e94a47a72ad8792b4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:35.121598 5447 tablet_bootstrap.cc:492] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:35.124881 5447 tablet_bootstrap.cc:654] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:35.125633 5210 tablet.cc:2369] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:35.133014 5360 tablet_bootstrap.cc:492] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:35.133236 5360 ts_tablet_manager.cc:1367] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.010s
I20240501 14:00:35.134033 5360 raft_consensus.cc:348] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.134318 5360 raft_consensus.cc:374] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.134409 5360 raft_consensus.cc:729] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.134716 5360 consensus_queue.cc:260] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.136003 5403 tablet_bootstrap.cc:492] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:35.136162 5403 ts_tablet_manager.cc:1367] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.008s
I20240501 14:00:35.137794 5403 raft_consensus.cc:348] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.138011 5403 raft_consensus.cc:374] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.138108 5403 raft_consensus.cc:729] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.138350 5403 consensus_queue.cc:260] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.138947 5403 ts_tablet_manager.cc:1398] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:00:35.139493 5360 ts_tablet_manager.cc:1398] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:35.140966 2002 catalog_manager.cc:5484] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:35.153489 5447 tablet_bootstrap.cc:492] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:35.153793 5447 ts_tablet_manager.cc:1367] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240501 14:00:35.154469 5447 raft_consensus.cc:348] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.154801 5447 raft_consensus.cc:374] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.154942 5447 raft_consensus.cc:729] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.156121 5447 consensus_queue.cc:260] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.162269 5447 ts_tablet_manager.cc:1398] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:35.172113 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:00:35.172396 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:00:35.176967 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 147d90e667bb41b69be550df94446f33
I20240501 14:00:35.177563 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet c52127afcd764e03835b00dab69ced7c
I20240501 14:00:35.177834 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 147d90e667bb41b69be550df94446f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:35.177968 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 147d90e667bb41b69be550df94446f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:35.178114 5390 tablet_replica.cc:330] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:35.178161 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 147d90e667bb41b69be550df94446f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:35.178316 5390 raft_consensus.cc:2229] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.178436 5391 tablet_replica.cc:330] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:35.178580 5390 raft_consensus.cc:2258] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.178670 5391 raft_consensus.cc:2229] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:35.179162 5391 raft_consensus.cc:2258] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.179301 5390 ts_tablet_manager.cc:1875] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.179847 5391 ts_tablet_manager.cc:1875] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.182695 5389 tablet_replica.cc:330] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:35.182929 5389 raft_consensus.cc:2229] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.183127 5389 raft_consensus.cc:2258] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.184326 5389 ts_tablet_manager.cc:1875] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.185808 5391 ts_tablet_manager.cc:1888] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.185940 5391 log.cc:1196] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/147d90e667bb41b69be550df94446f33
I20240501 14:00:35.186337 5391 ts_tablet_manager.cc:1909] T 147d90e667bb41b69be550df94446f33 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:35.186967 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet c52127afcd764e03835b00dab69ced7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:35.187467 5391 tablet_replica.cc:330] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:35.187564 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 147d90e667bb41b69be550df94446f33 (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.187773 5391 raft_consensus.cc:2229] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:35.188148 5391 raft_consensus.cc:2258] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.188350 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet c52127afcd764e03835b00dab69ced7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:35.188632 5392 tablet_replica.cc:330] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:35.188822 5392 raft_consensus.cc:2229] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.189049 5392 raft_consensus.cc:2258] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.189092 5391 ts_tablet_manager.cc:1875] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.189759 5392 ts_tablet_manager.cc:1875] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.190742 5389 ts_tablet_manager.cc:1888] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.190920 5389 log.cc:1196] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/147d90e667bb41b69be550df94446f33
I20240501 14:00:35.191321 5389 ts_tablet_manager.cc:1909] T 147d90e667bb41b69be550df94446f33 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:35.192970 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 147d90e667bb41b69be550df94446f33 (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.195626 5391 ts_tablet_manager.cc:1888] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.195848 5391 log.cc:1196] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/c52127afcd764e03835b00dab69ced7c
I20240501 14:00:35.196352 5391 ts_tablet_manager.cc:1909] T c52127afcd764e03835b00dab69ced7c P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:35.196625 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 809ef5a337774c8d98aea88a79e5152e
I20240501 14:00:35.197086 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet c52127afcd764e03835b00dab69ced7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:35.197484 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 809ef5a337774c8d98aea88a79e5152e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:35.197603 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 809ef5a337774c8d98aea88a79e5152e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:35.197952 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 809ef5a337774c8d98aea88a79e5152e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:35.198587 5391 tablet_replica.cc:330] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:35.198731 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet c52127afcd764e03835b00dab69ced7c (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.198802 5389 tablet_replica.cc:330] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:35.199000 5389 raft_consensus.cc:2229] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:35.199384 5389 raft_consensus.cc:2258] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.199764 5390 ts_tablet_manager.cc:1888] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.199889 5390 log.cc:1196] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/147d90e667bb41b69be550df94446f33
I20240501 14:00:35.200143 5389 ts_tablet_manager.cc:1875] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.200335 5390 ts_tablet_manager.cc:1909] T 147d90e667bb41b69be550df94446f33 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:35.200582 4961 raft_consensus.cc:482] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:35.200722 4961 raft_consensus.cc:504] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:35.201272 5390 tablet_replica.cc:330] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:35.201401 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 147d90e667bb41b69be550df94446f33 (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.201498 5390 raft_consensus.cc:2229] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.201663 4961 leader_election.cc:288] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:35.202006 5392 ts_tablet_manager.cc:1888] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.202121 5392 log.cc:1196] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/c52127afcd764e03835b00dab69ced7c
I20240501 14:00:35.202566 5392 ts_tablet_manager.cc:1909] T c52127afcd764e03835b00dab69ced7c P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:35.203589 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet c52127afcd764e03835b00dab69ced7c (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.206147 5389 ts_tablet_manager.cc:1888] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.206557 5391 raft_consensus.cc:2229] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.206902 5391 raft_consensus.cc:2258] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.207877 5391 ts_tablet_manager.cc:1875] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.214679 5389 log.cc:1196] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/809ef5a337774c8d98aea88a79e5152e
I20240501 14:00:35.215304 5389 ts_tablet_manager.cc:1909] T 809ef5a337774c8d98aea88a79e5152e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:35.216521 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 809ef5a337774c8d98aea88a79e5152e (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.216765 5391 ts_tablet_manager.cc:1888] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.216984 5391 log.cc:1196] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/809ef5a337774c8d98aea88a79e5152e
I20240501 14:00:35.217504 5391 ts_tablet_manager.cc:1909] T 809ef5a337774c8d98aea88a79e5152e P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:35.217833 5409 raft_consensus.cc:482] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:35.218039 5409 raft_consensus.cc:504] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.220228 5409 leader_election.cc:288] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:35.201725 5390 raft_consensus.cc:2258] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.222299 5390 ts_tablet_manager.cc:1875] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.226728 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 809ef5a337774c8d98aea88a79e5152e (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.234252 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa55eb93e93c4c01a4353d20d7fac6ea" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:35.234791 4901 raft_consensus.cc:2454] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:35.235785 4968 leader_election.cc:302] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:35.236475 5409 raft_consensus.cc:2781] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:35.236613 5409 raft_consensus.cc:482] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:35.236698 5409 raft_consensus.cc:3037] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.241000 5409 raft_consensus.cc:504] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.242583 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa55eb93e93c4c01a4353d20d7fac6ea" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:35.242826 4901 raft_consensus.cc:3037] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.245829 4901 raft_consensus.cc:2454] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:35.246430 4968 leader_election.cc:302] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:35.249589 5365 raft_consensus.cc:482] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:35.249805 5365 raft_consensus.cc:504] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:35.250816 5409 leader_election.cc:288] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:35.251051 5409 raft_consensus.cc:2781] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:35.251215 5409 raft_consensus.cc:686] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:35.251721 5409 consensus_queue.cc:237] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:35.252449 5365 leader_election.cc:288] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:35.276167 5390 ts_tablet_manager.cc:1888] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.276494 5390 log.cc:1196] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/c52127afcd764e03835b00dab69ced7c
I20240501 14:00:35.277009 5390 ts_tablet_manager.cc:1909] T c52127afcd764e03835b00dab69ced7c P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:35.277688 5408 raft_consensus.cc:482] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:35.277585 2003 catalog_manager.cc:5484] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:35.277890 5408 raft_consensus.cc:504] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:35.278395 5390 tablet_replica.cc:330] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:35.278699 5408 leader_election.cc:288] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:35.278712 5390 raft_consensus.cc:2229] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.278772 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet c52127afcd764e03835b00dab69ced7c (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.279098 5390 raft_consensus.cc:2258] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.280661 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:35.280936 5164 raft_consensus.cc:2454] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:35.282791 2471 leader_election.cc:302] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:35.283123 5426 raft_consensus.cc:2781] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:35.283227 5426 raft_consensus.cc:482] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:35.283345 5426 raft_consensus.cc:3037] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.285532 5390 ts_tablet_manager.cc:1875] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.288396 5426 raft_consensus.cc:504] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:35.296391 5426 leader_election.cc:288] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:35.297930 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:35.298233 5164 raft_consensus.cc:3037] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.301486 5164 raft_consensus.cc:2454] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:35.302027 2471 leader_election.cc:302] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:35.302407 5426 raft_consensus.cc:2781] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:35.302598 5426 raft_consensus.cc:686] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:35.302857 5426 consensus_queue.cc:237] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
W20240501 14:00:35.305544 2454 tablet.cc:2369] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:35.306942 1999 catalog_manager.cc:5484] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:35.312203 5390 ts_tablet_manager.cc:1888] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.312436 5390 log.cc:1196] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/809ef5a337774c8d98aea88a79e5152e
I20240501 14:00:35.313206 5390 ts_tablet_manager.cc:1909] T 809ef5a337774c8d98aea88a79e5152e P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:35.314251 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa55eb93e93c4c01a4353d20d7fac6ea" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:35.314476 2406 raft_consensus.cc:3037] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.317173 2406 raft_consensus.cc:2454] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:35.319864 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa55eb93e93c4c01a4353d20d7fac6ea" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:35.320163 2407 raft_consensus.cc:2362] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1. Re-sending same reply.
I20240501 14:00:35.328078 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:35.328397 5296 raft_consensus.cc:2454] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:35.328584 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:35.328847 5099 leader_election.cc:302] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:35.329195 5408 raft_consensus.cc:2741] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 14:00:35.329964 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:35.330179 5296 raft_consensus.cc:2454] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:35.330687 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:35.330880 5296 raft_consensus.cc:3037] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:35.331912 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 809ef5a337774c8d98aea88a79e5152e (table rebalance_test_table_2 [id=309191e091764d0298f1f6d4964df36c]) successfully deleted
I20240501 14:00:35.333422 5296 raft_consensus.cc:2454] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:35.334118 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:35.334333 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052964af500c427e836f0f3121aa91f3" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:35.334458 5164 raft_consensus.cc:2379] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a4b9a96e9734baa8a03d6056bfa934c in current term 1: Already voted for candidate 745ac12076e84effa5bde1ecbd6a32ec in this term.
I20240501 14:00:35.335770 5228 leader_election.cc:302] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:35.336313 5365 raft_consensus.cc:2726] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:35.362828 5275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:35.364270 5014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:35.368317 2517 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:00:35.370095 5408 consensus_queue.cc:1035] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.411588 2388 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:00:35.412432 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:35.423023 5425 consensus_queue.cc:1035] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.452097 5144 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:35.463650 4901 raft_consensus.cc:1261] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:35.464675 2406 raft_consensus.cc:1261] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:35.466064 5409 consensus_queue.cc:1035] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.468813 5409 consensus_queue.cc:1035] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.488584 1999 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet aa55eb93e93c4c01a4353d20d7fac6ea (table rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602]). Expected version 1 got 0
I20240501 14:00:35.495747 2003 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet aa55eb93e93c4c01a4353d20d7fac6ea (table rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602]). Expected version 1 got 0
I20240501 14:00:35.505192 5492 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:35.510424 5486 mvcc.cc:205] Tried to move back new op lower bound from 7022887057248432128 to 7022887056395874304. Current Snapshot: MvccSnapshot[applied={T|T < 7022887057248432128}]
I20240501 14:00:35.513172 1999 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet aa55eb93e93c4c01a4353d20d7fac6ea (table rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602]). Expected version 1 got 0
I20240501 14:00:35.521082 5364 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.521643 2125 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:00:35.521984 2256 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:00:35.524394 5491 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:35.535890 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602] alter complete (version 1)
I20240501 14:00:35.538653 5364 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.539651 5492 tablet.cc:1620] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:35.540910 5491 tablet.cc:1620] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:35.570983 5493 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:35.573554 5493 tablet.cc:1620] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:35.588289 1999 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet aa55eb93e93c4c01a4353d20d7fac6ea (table rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602]). Expected version 1 got 0
I20240501 14:00:35.592480 4881 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:35.651234 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:35.661825 5491 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:35.662379 5491 tablet.cc:1620] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:35.662673 5493 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:35.663007 5492 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:35.663130 5493 tablet.cc:1620] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:35.663491 5492 tablet.cc:1620] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:35.680500 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=181a12c7e92f41fa9926bd332fd34602] alter complete (version 2)
I20240501 14:00:35.699046 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:00:35.699344 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:00:35.703635 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 0b461bde7a84495f8828c69001a42944
I20240501 14:00:35.704622 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b461bde7a84495f8828c69001a42944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:35.704907 5392 tablet_replica.cc:330] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:35.705121 5392 raft_consensus.cc:2229] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.705346 5392 raft_consensus.cc:2258] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.706090 5392 ts_tablet_manager.cc:1875] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.709300 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b461bde7a84495f8828c69001a42944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:35.709858 5389 tablet_replica.cc:330] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:35.710083 5389 raft_consensus.cc:2229] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.710296 5389 raft_consensus.cc:2258] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.711035 5389 ts_tablet_manager.cc:1875] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.720840 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 504e8f4db6434b68a53dd52e08535090
I20240501 14:00:35.721069 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b461bde7a84495f8828c69001a42944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:35.721470 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet bd0cfe423a214238ab93a6131545cef0
I20240501 14:00:35.721797 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 504e8f4db6434b68a53dd52e08535090 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:35.721884 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 504e8f4db6434b68a53dd52e08535090 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:35.722174 5390 tablet_replica.cc:330] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:35.722364 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet bd0cfe423a214238ab93a6131545cef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:35.722424 5390 raft_consensus.cc:2229] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.722738 5390 raft_consensus.cc:2258] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.723009 5392 ts_tablet_manager.cc:1888] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.723177 5392 log.cc:1196] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/0b461bde7a84495f8828c69001a42944
I20240501 14:00:35.723294 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet bd0cfe423a214238ab93a6131545cef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:35.723481 5390 ts_tablet_manager.cc:1875] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.723661 5392 ts_tablet_manager.cc:1909] T 0b461bde7a84495f8828c69001a42944 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:35.723824 5389 ts_tablet_manager.cc:1888] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.723960 5389 log.cc:1196] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b461bde7a84495f8828c69001a42944
I20240501 14:00:35.724728 5392 tablet_replica.cc:330] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:35.724943 5392 raft_consensus.cc:2229] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.725003 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 0b461bde7a84495f8828c69001a42944 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.725250 5392 raft_consensus.cc:2258] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.725570 5389 ts_tablet_manager.cc:1909] T 0b461bde7a84495f8828c69001a42944 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:35.725942 5392 ts_tablet_manager.cc:1875] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.726601 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 504e8f4db6434b68a53dd52e08535090 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:35.726907 5389 tablet_replica.cc:330] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:35.727082 5389 raft_consensus.cc:2229] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.727304 5389 raft_consensus.cc:2258] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.727948 5389 ts_tablet_manager.cc:1875] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.730144 5390 ts_tablet_manager.cc:1888] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.730307 5390 log.cc:1196] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/504e8f4db6434b68a53dd52e08535090
I20240501 14:00:35.730966 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet bd0cfe423a214238ab93a6131545cef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:35.731791 5390 ts_tablet_manager.cc:1909] T 504e8f4db6434b68a53dd52e08535090 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:35.732475 5392 ts_tablet_manager.cc:1888] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.732597 5392 log.cc:1196] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd0cfe423a214238ab93a6131545cef0
I20240501 14:00:35.733536 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 0b461bde7a84495f8828c69001a42944 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
W20240501 14:00:35.733808 2080 consensus_peers.cc:485] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b461bde7a84495f8828c69001a42944. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:35.733861 5392 ts_tablet_manager.cc:1909] T bd0cfe423a214238ab93a6131545cef0 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:35.734915 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet bd0cfe423a214238ab93a6131545cef0 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.735929 5389 ts_tablet_manager.cc:1888] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.736039 5389 log.cc:1196] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/504e8f4db6434b68a53dd52e08535090
I20240501 14:00:35.736405 5389 ts_tablet_manager.cc:1909] T 504e8f4db6434b68a53dd52e08535090 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:35.737386 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 504e8f4db6434b68a53dd52e08535090 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.738595 5508 tablet_replica.cc:330] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:35.738847 5508 raft_consensus.cc:2229] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:35.739241 5508 raft_consensus.cc:2258] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.740072 5508 ts_tablet_manager.cc:1875] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.743791 5390 tablet_replica.cc:330] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:35.744061 5390 raft_consensus.cc:2229] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.744467 5390 raft_consensus.cc:2258] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.744668 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 504e8f4db6434b68a53dd52e08535090 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.745297 5390 ts_tablet_manager.cc:1875] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.749378 5508 ts_tablet_manager.cc:1888] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.750255 5508 log.cc:1196] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b461bde7a84495f8828c69001a42944
I20240501 14:00:35.750845 5508 ts_tablet_manager.cc:1909] T 0b461bde7a84495f8828c69001a42944 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:35.752171 5508 tablet_replica.cc:330] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:35.752261 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 0b461bde7a84495f8828c69001a42944 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.752982 5508 raft_consensus.cc:2229] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:35.753545 5508 raft_consensus.cc:2258] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.754814 5508 ts_tablet_manager.cc:1875] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.758306 5390 ts_tablet_manager.cc:1888] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.759055 5390 log.cc:1196] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/bd0cfe423a214238ab93a6131545cef0
I20240501 14:00:35.759608 5390 ts_tablet_manager.cc:1909] T bd0cfe423a214238ab93a6131545cef0 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:35.760704 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet bd0cfe423a214238ab93a6131545cef0 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.762897 5508 ts_tablet_manager.cc:1888] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.763126 5508 log.cc:1196] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/504e8f4db6434b68a53dd52e08535090
I20240501 14:00:35.764895 5508 ts_tablet_manager.cc:1909] T 504e8f4db6434b68a53dd52e08535090 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:35.765889 5508 tablet_replica.cc:330] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:35.766178 5508 raft_consensus.cc:2229] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:35.766657 5508 raft_consensus.cc:2258] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.766765 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 504e8f4db6434b68a53dd52e08535090 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.767524 5508 ts_tablet_manager.cc:1875] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.774183 5508 ts_tablet_manager.cc:1888] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:35.774441 5508 log.cc:1196] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd0cfe423a214238ab93a6131545cef0
I20240501 14:00:35.774896 5508 ts_tablet_manager.cc:1909] T bd0cfe423a214238ab93a6131545cef0 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:35.775759 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet bd0cfe423a214238ab93a6131545cef0 (table rebalance_test_table_4 [id=8c306a4efe6d41968488d3146cdfb987]) successfully deleted
I20240501 14:00:35.788307 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:35.797259 5491 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:35.802562 5493 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:35.803426 5492 tablet.cc:1722] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:35.816469 5426 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.818782 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=181a12c7e92f41fa9926bd332fd34602] alter complete (version 3)
I20240501 14:00:35.829229 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:35.841217 2125 tablet_service.cc:1467] Processing CreateTablet for tablet d5086890bd254edebdf12812ff5c3728 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.841789 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5086890bd254edebdf12812ff5c3728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.841953 5014 tablet_service.cc:1467] Processing CreateTablet for tablet d5086890bd254edebdf12812ff5c3728 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.842145 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 44a4e92b144f47b5800506f4bc3eadb6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.842571 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5086890bd254edebdf12812ff5c3728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.842702 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4e92b144f47b5800506f4bc3eadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.846983 4881 tablet_service.cc:1467] Processing CreateTablet for tablet d5086890bd254edebdf12812ff5c3728 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.847502 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5086890bd254edebdf12812ff5c3728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.847450 2126 tablet_service.cc:1467] Processing CreateTablet for tablet c685294d3fca4677bc1b1cf355dac3ce (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.847829 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c685294d3fca4677bc1b1cf355dac3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.851166 4880 tablet_service.cc:1467] Processing CreateTablet for tablet c685294d3fca4677bc1b1cf355dac3ce (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.851660 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c685294d3fca4677bc1b1cf355dac3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.852430 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 44a4e92b144f47b5800506f4bc3eadb6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.852880 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4e92b144f47b5800506f4bc3eadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.854709 4961 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.855180 5013 tablet_service.cc:1467] Processing CreateTablet for tablet c685294d3fca4677bc1b1cf355dac3ce (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.855664 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c685294d3fca4677bc1b1cf355dac3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.859035 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 44a4e92b144f47b5800506f4bc3eadb6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:35.859510 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4e92b144f47b5800506f4bc3eadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:35.914095 5528 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:35.915213 5524 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:35.917358 5524 tablet_bootstrap.cc:654] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.918995 5527 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:35.922406 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:35.922688 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:35.924089 5525 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:35.926321 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet aa55eb93e93c4c01a4353d20d7fac6ea
I20240501 14:00:35.927603 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet aa55eb93e93c4c01a4353d20d7fac6ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:35.927876 5392 tablet_replica.cc:330] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:35.928087 5392 raft_consensus.cc:2229] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.928236 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet aa55eb93e93c4c01a4353d20d7fac6ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:35.928303 5392 raft_consensus.cc:2258] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.928335 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet aa55eb93e93c4c01a4353d20d7fac6ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:35 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:35.929486 5392 ts_tablet_manager.cc:1875] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.936332 5392 ts_tablet_manager.cc:1888] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:35.936618 5392 log.cc:1196] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa55eb93e93c4c01a4353d20d7fac6ea
I20240501 14:00:35.937142 5392 ts_tablet_manager.cc:1909] T aa55eb93e93c4c01a4353d20d7fac6ea P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:35.938228 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet aa55eb93e93c4c01a4353d20d7fac6ea (table rebalancer_dynamic_table_new_name [id=181a12c7e92f41fa9926bd332fd34602]) successfully deleted
I20240501 14:00:35.942639 5528 tablet_bootstrap.cc:654] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.943706 5525 tablet_bootstrap.cc:654] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.946295 5530 tablet_replica.cc:330] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:35.946600 5530 raft_consensus.cc:2229] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:35.946791 5522 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:35.947001 5530 raft_consensus.cc:2258] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.947990 5530 ts_tablet_manager.cc:1875] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.953296 5527 tablet_bootstrap.cc:654] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.943854 5531 tablet_replica.cc:330] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:35.962615 5531 raft_consensus.cc:2229] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:35.963304 5531 raft_consensus.cc:2258] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:35.965090 5531 ts_tablet_manager.cc:1875] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:35.967208 5524 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:35.967367 5524 ts_tablet_manager.cc:1367] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240501 14:00:35.968710 5522 tablet_bootstrap.cc:654] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.968729 5528 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:35.968923 5528 ts_tablet_manager.cc:1367] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240501 14:00:35.970048 5528 raft_consensus.cc:348] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:35.970238 5528 raft_consensus.cc:374] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.970201 5524 raft_consensus.cc:348] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.970350 5528 raft_consensus.cc:729] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.970418 5524 raft_consensus.cc:374] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.970557 5524 raft_consensus.cc:729] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.970746 5528 consensus_queue.cc:260] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:35.970849 5524 consensus_queue.cc:260] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.971521 5524 ts_tablet_manager.cc:1398] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:35.972018 5528 ts_tablet_manager.cc:1398] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:35.972893 5528 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:35.975122 5531 ts_tablet_manager.cc:1888] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:35.978166 5530 ts_tablet_manager.cc:1888] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:35.978433 5530 log.cc:1196] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/aa55eb93e93c4c01a4353d20d7fac6ea
I20240501 14:00:35.979004 5530 ts_tablet_manager.cc:1909] T aa55eb93e93c4c01a4353d20d7fac6ea P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:35.980859 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet aa55eb93e93c4c01a4353d20d7fac6ea (table rebalancer_dynamic_table_new_name [id=181a12c7e92f41fa9926bd332fd34602]) successfully deleted
I20240501 14:00:35.976230 5528 tablet_bootstrap.cc:654] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.985970 5531 log.cc:1196] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/aa55eb93e93c4c01a4353d20d7fac6ea
I20240501 14:00:35.986765 5531 ts_tablet_manager.cc:1909] T aa55eb93e93c4c01a4353d20d7fac6ea P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:35.988881 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet aa55eb93e93c4c01a4353d20d7fac6ea (table rebalancer_dynamic_table_new_name [id=181a12c7e92f41fa9926bd332fd34602]) successfully deleted
I20240501 14:00:35.989414 5524 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:35.990252 5522 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:35.990429 5522 ts_tablet_manager.cc:1367] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240501 14:00:35.991009 5525 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:35.991109 5524 tablet_bootstrap.cc:654] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:35.991169 5525 ts_tablet_manager.cc:1367] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.016s
I20240501 14:00:35.991276 5522 raft_consensus.cc:348] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.991497 5522 raft_consensus.cc:374] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.991572 5522 raft_consensus.cc:729] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.991801 5522 consensus_queue.cc:260] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.991961 5525 raft_consensus.cc:348] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.992177 5525 raft_consensus.cc:374] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:35.992256 5525 raft_consensus.cc:729] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:35.992292 5522 ts_tablet_manager.cc:1398] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:35.992673 5525 consensus_queue.cc:260] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:35.996143 5525 ts_tablet_manager.cc:1398] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:36.016943 5527 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:36.017226 5527 ts_tablet_manager.cc:1367] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.006s
I20240501 14:00:36.018132 5527 raft_consensus.cc:348] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.018472 5527 raft_consensus.cc:374] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.018627 5527 raft_consensus.cc:729] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.018962 5527 consensus_queue.cc:260] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.026368 5527 ts_tablet_manager.cc:1398] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:00:36.030193 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:36.032419 5527 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:36.034375 5527 tablet_bootstrap.cc:654] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.040859 5275 tablet_service.cc:1467] Processing CreateTablet for tablet dba319e59aa84691a3e4ccbb4c365870 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee]), partition=
I20240501 14:00:36.041095 5013 tablet_service.cc:1467] Processing CreateTablet for tablet dba319e59aa84691a3e4ccbb4c365870 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee]), partition=
I20240501 14:00:36.041287 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba319e59aa84691a3e4ccbb4c365870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.041517 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba319e59aa84691a3e4ccbb4c365870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.046864 4881 tablet_service.cc:1467] Processing CreateTablet for tablet dba319e59aa84691a3e4ccbb4c365870 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee]), partition=
I20240501 14:00:36.047289 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba319e59aa84691a3e4ccbb4c365870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.052026 5522 tablet_bootstrap.cc:492] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:36.053803 5522 tablet_bootstrap.cc:654] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.054353 5528 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:36.054579 5528 ts_tablet_manager.cc:1367] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.082s user 0.018s sys 0.008s
I20240501 14:00:36.055578 5528 raft_consensus.cc:348] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.055873 5528 raft_consensus.cc:374] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.055969 5528 raft_consensus.cc:729] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.056970 5528 consensus_queue.cc:260] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.057669 5528 ts_tablet_manager.cc:1398] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:36.058311 5528 tablet_bootstrap.cc:492] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:36.060194 5528 tablet_bootstrap.cc:654] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.070057 5524 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:36.070362 5524 ts_tablet_manager.cc:1367] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.081s user 0.019s sys 0.008s
I20240501 14:00:36.071269 5524 raft_consensus.cc:348] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.071568 5524 raft_consensus.cc:374] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.071691 5524 raft_consensus.cc:729] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.071983 5524 consensus_queue.cc:260] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.072635 5524 ts_tablet_manager.cc:1398] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:36.074105 5524 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:36.080545 5524 tablet_bootstrap.cc:654] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.084741 5527 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:36.084985 5527 ts_tablet_manager.cc:1367] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.000s
I20240501 14:00:36.086166 5527 raft_consensus.cc:348] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.086432 5527 raft_consensus.cc:374] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.086558 5527 raft_consensus.cc:729] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.086833 5527 consensus_queue.cc:260] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.087424 5527 ts_tablet_manager.cc:1398] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:36.088971 5528 tablet_bootstrap.cc:492] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:36.089177 5528 ts_tablet_manager.cc:1367] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.021s
I20240501 14:00:36.089208 5527 tablet_bootstrap.cc:492] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:36.089548 5425 raft_consensus.cc:482] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.089728 5425 raft_consensus.cc:504] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:36.089995 5528 raft_consensus.cc:348] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.090190 5528 raft_consensus.cc:374] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.090273 5528 raft_consensus.cc:729] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.090549 5528 consensus_queue.cc:260] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.090652 5425 leader_election.cc:288] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:36.090896 5527 tablet_bootstrap.cc:654] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.091075 5528 ts_tablet_manager.cc:1398] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:36.092257 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4e92b144f47b5800506f4bc3eadb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:36.092504 5296 raft_consensus.cc:2454] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:36.094178 5099 leader_election.cc:302] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:36.096772 5524 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:36.096966 5524 ts_tablet_manager.cc:1367] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 14:00:36.098115 5524 raft_consensus.cc:348] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.098368 5524 raft_consensus.cc:374] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.098462 5524 raft_consensus.cc:729] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.098727 5524 consensus_queue.cc:260] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.099403 5524 ts_tablet_manager.cc:1398] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:36.100733 5527 tablet_bootstrap.cc:492] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:36.100929 5527 ts_tablet_manager.cc:1367] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:00:36.101579 5425 raft_consensus.cc:2781] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:36.101694 5425 raft_consensus.cc:482] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:36.101794 5425 raft_consensus.cc:3037] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.101799 5527 raft_consensus.cc:348] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.101982 5527 raft_consensus.cc:374] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.102066 5527 raft_consensus.cc:729] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.102290 5527 consensus_queue.cc:260] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.102900 5527 ts_tablet_manager.cc:1398] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:36.105423 5522 tablet_bootstrap.cc:492] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:36.105618 5522 ts_tablet_manager.cc:1367] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.000s
I20240501 14:00:36.106403 5522 raft_consensus.cc:348] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.106750 5522 raft_consensus.cc:374] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.106863 5522 raft_consensus.cc:729] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.107169 5522 consensus_queue.cc:260] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.107780 5522 ts_tablet_manager.cc:1398] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:36.113595 5425 raft_consensus.cc:504] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:36.115136 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4e92b144f47b5800506f4bc3eadb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:36.115376 5296 raft_consensus.cc:3037] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.118458 5296 raft_consensus.cc:2454] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:36.123155 5099 leader_election.cc:302] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:36.123593 5408 raft_consensus.cc:2781] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.127460 5425 leader_election.cc:288] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:36.130322 5464 raft_consensus.cc:482] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.131481 5464 raft_consensus.cc:504] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.132555 5464 leader_election.cc:288] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:36.134544 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5086890bd254edebdf12812ff5c3728" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:36.134817 4901 raft_consensus.cc:2454] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:36.136159 4968 leader_election.cc:302] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:36.136484 5409 raft_consensus.cc:482] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.136639 5409 raft_consensus.cc:504] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.137478 5409 leader_election.cc:288] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:36.139751 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba319e59aa84691a3e4ccbb4c365870" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:36.140019 4901 raft_consensus.cc:2454] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:36.141924 4968 leader_election.cc:302] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:36.145279 5409 raft_consensus.cc:2781] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:36.145478 5409 raft_consensus.cc:482] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:36.145583 5409 raft_consensus.cc:3037] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.152673 5446 raft_consensus.cc:482] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.152849 5446 raft_consensus.cc:504] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:36.153857 5446 leader_election.cc:288] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:36.157805 5464 raft_consensus.cc:2781] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:36.157954 5464 raft_consensus.cc:482] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:36.158035 5464 raft_consensus.cc:3037] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.163414 5409 raft_consensus.cc:504] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.161403 5464 raft_consensus.cc:504] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.164543 5464 leader_election.cc:288] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:36.164575 5409 leader_election.cc:288] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:36.164654 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5086890bd254edebdf12812ff5c3728" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:36.164866 4901 raft_consensus.cc:3037] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.169525 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba319e59aa84691a3e4ccbb4c365870" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:36.169873 4900 raft_consensus.cc:3037] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.177311 4900 raft_consensus.cc:2454] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:36.178315 4968 leader_election.cc:302] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:36.180940 4901 raft_consensus.cc:2454] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:36.182191 4968 leader_election.cc:302] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:36.183923 5409 raft_consensus.cc:2781] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.184142 5464 raft_consensus.cc:2781] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.190552 5408 raft_consensus.cc:686] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:36.190922 5408 consensus_queue.cc:237] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:36.194937 5562 raft_consensus.cc:482] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.195880 5562 raft_consensus.cc:504] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.197521 5464 raft_consensus.cc:686] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:36.197837 5464 consensus_queue.cc:237] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.201473 5409 raft_consensus.cc:686] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:36.202273 5409 consensus_queue.cc:237] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.206518 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba319e59aa84691a3e4ccbb4c365870" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:36.206804 5296 raft_consensus.cc:2454] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:36.206665 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba319e59aa84691a3e4ccbb4c365870" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:36.207486 5295 raft_consensus.cc:3037] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.208374 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c685294d3fca4677bc1b1cf355dac3ce" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:36.208621 4901 raft_consensus.cc:2454] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:36.209156 4968 leader_election.cc:302] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:36.209502 5560 raft_consensus.cc:2781] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:36.209626 5560 raft_consensus.cc:482] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:36.209810 5560 raft_consensus.cc:3037] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.211125 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5086890bd254edebdf12812ff5c3728" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:36.211344 2146 raft_consensus.cc:3037] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.212898 5560 raft_consensus.cc:504] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.214082 2146 raft_consensus.cc:2454] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:36.214601 5560 leader_election.cc:288] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:36.214926 5562 leader_election.cc:288] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:36.215008 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c685294d3fca4677bc1b1cf355dac3ce" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:36.215243 4901 raft_consensus.cc:3037] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.216136 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4e92b144f47b5800506f4bc3eadb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:36.216408 2143 raft_consensus.cc:3037] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.216589 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4e92b144f47b5800506f4bc3eadb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:36.216820 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c685294d3fca4677bc1b1cf355dac3ce" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:36.216982 2144 raft_consensus.cc:2454] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:36.217845 4901 raft_consensus.cc:2454] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:36.217872 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5086890bd254edebdf12812ff5c3728" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:36.218051 2147 raft_consensus.cc:2362] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1. Re-sending same reply.
I20240501 14:00:36.219367 4968 leader_election.cc:302] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:36.219686 5562 raft_consensus.cc:2781] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.219789 2143 raft_consensus.cc:2454] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:36.221114 2002 catalog_manager.cc:5484] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:36.222519 5562 raft_consensus.cc:686] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:36.222648 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4e92b144f47b5800506f4bc3eadb6" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:36.222955 5296 raft_consensus.cc:2379] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1a414d381f44829a2510cb76d481887 in current term 1: Already voted for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in this term.
I20240501 14:00:36.223443 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c685294d3fca4677bc1b1cf355dac3ce" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:36.223116 5562 consensus_queue.cc:237] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:36.224189 1999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:00:36.224395 1999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:00:36.223634 2146 raft_consensus.cc:3037] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.227192 5295 raft_consensus.cc:2454] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:36.228899 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet c60e9cd1e3a748aeb480df2ee8d97ba3
I20240501 14:00:36.229462 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 9b7498ef46b94171bcd84fc4ebfc3ae5
I20240501 14:00:36.229815 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet e16034fbf5ce43cebd5d1d3a217b79b3
I20240501 14:00:36.230859 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet c60e9cd1e3a748aeb480df2ee8d97ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:36.230908 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet c60e9cd1e3a748aeb480df2ee8d97ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:36.231078 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:36.231331 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet e16034fbf5ce43cebd5d1d3a217b79b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:36.231436 5390 tablet_replica.cc:330] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:36.231462 5389 tablet_replica.cc:330] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:36.231688 5390 raft_consensus.cc:2229] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.231688 5389 raft_consensus.cc:2229] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:36.231695 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet c60e9cd1e3a748aeb480df2ee8d97ba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:36.231910 5390 raft_consensus.cc:2258] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.231966 5508 tablet_replica.cc:330] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:36.232045 5389 raft_consensus.cc:2258] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.232137 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:36.232180 5508 raft_consensus.cc:2229] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.232403 5508 raft_consensus.cc:2258] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.232578 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet e16034fbf5ce43cebd5d1d3a217b79b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:36.232666 5390 ts_tablet_manager.cc:1875] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.232767 5389 ts_tablet_manager.cc:1875] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.233038 5508 ts_tablet_manager.cc:1875] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.234706 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:36.235008 5392 tablet_replica.cc:330] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:36.235230 5392 raft_consensus.cc:2229] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.235488 5392 raft_consensus.cc:2258] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.236245 5392 ts_tablet_manager.cc:1875] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.239406 5508 ts_tablet_manager.cc:1888] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.239647 5508 log.cc:1196] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/c60e9cd1e3a748aeb480df2ee8d97ba3
I20240501 14:00:36.239886 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet e16034fbf5ce43cebd5d1d3a217b79b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:36.240128 5508 ts_tablet_manager.cc:1909] T c60e9cd1e3a748aeb480df2ee8d97ba3 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:36.240852 5390 ts_tablet_manager.cc:1888] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.241015 5390 log.cc:1196] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/c60e9cd1e3a748aeb480df2ee8d97ba3
I20240501 14:00:36.241272 5508 tablet_replica.cc:330] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:36.241444 5390 ts_tablet_manager.cc:1909] T c60e9cd1e3a748aeb480df2ee8d97ba3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:36.241202 1999 catalog_manager.cc:5484] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:36.241549 5508 raft_consensus.cc:2229] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:36.241981 5508 raft_consensus.cc:2258] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.243477 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet c60e9cd1e3a748aeb480df2ee8d97ba3 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.243690 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet c60e9cd1e3a748aeb480df2ee8d97ba3 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.246343 5389 ts_tablet_manager.cc:1888] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.246572 5389 log.cc:1196] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/c60e9cd1e3a748aeb480df2ee8d97ba3
I20240501 14:00:36.247015 5389 ts_tablet_manager.cc:1909] T c60e9cd1e3a748aeb480df2ee8d97ba3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:36.247671 5508 ts_tablet_manager.cc:1875] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.248211 5389 tablet_replica.cc:330] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:36.248497 5389 raft_consensus.cc:2229] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:36.248260 2002 catalog_manager.cc:5484] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:36.248901 5389 raft_consensus.cc:2258] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.248869 2002 catalog_manager.cc:5484] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:36.249337 5392 ts_tablet_manager.cc:1888] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.245764 2146 raft_consensus.cc:2454] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:36.249732 5392 log.cc:1196] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b7498ef46b94171bcd84fc4ebfc3ae5
I20240501 14:00:36.250198 5392 ts_tablet_manager.cc:1909] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:36.250488 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet c60e9cd1e3a748aeb480df2ee8d97ba3 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.249811 5389 ts_tablet_manager.cc:1875] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.251467 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.251776 5390 tablet_replica.cc:330] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:36.252020 5390 raft_consensus.cc:2229] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.252322 5390 raft_consensus.cc:2258] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.254557 5508 ts_tablet_manager.cc:1888] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.255328 5508 log.cc:1196] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b7498ef46b94171bcd84fc4ebfc3ae5
I20240501 14:00:36.255868 5508 ts_tablet_manager.cc:1909] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:36.255920 5390 ts_tablet_manager.cc:1875] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.256934 5508 tablet_replica.cc:330] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:36.257010 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.257280 5508 raft_consensus.cc:2229] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.257546 5508 raft_consensus.cc:2258] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.258342 5508 ts_tablet_manager.cc:1875] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.258046 5389 ts_tablet_manager.cc:1888] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.261462 5389 log.cc:1196] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/e16034fbf5ce43cebd5d1d3a217b79b3
I20240501 14:00:36.261917 5389 ts_tablet_manager.cc:1909] T e16034fbf5ce43cebd5d1d3a217b79b3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:36.262990 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet e16034fbf5ce43cebd5d1d3a217b79b3 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.265050 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a4e92b144f47b5800506f4bc3eadb6" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:36.265739 2080 leader_election.cc:302] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:36.266047 4760 raft_consensus.cc:2726] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:36.267549 5390 ts_tablet_manager.cc:1888] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.267684 5390 log.cc:1196] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/9b7498ef46b94171bcd84fc4ebfc3ae5
I20240501 14:00:36.268102 5390 ts_tablet_manager.cc:1909] T 9b7498ef46b94171bcd84fc4ebfc3ae5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:36.268564 5508 ts_tablet_manager.cc:1888] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.268762 5508 log.cc:1196] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/e16034fbf5ce43cebd5d1d3a217b79b3
I20240501 14:00:36.269184 5508 ts_tablet_manager.cc:1909] T e16034fbf5ce43cebd5d1d3a217b79b3 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:36.269141 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 9b7498ef46b94171bcd84fc4ebfc3ae5 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.269415 5390 tablet_replica.cc:330] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:36.269655 5390 raft_consensus.cc:2229] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.270179 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet e16034fbf5ce43cebd5d1d3a217b79b3 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.270409 5390 raft_consensus.cc:2258] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.271301 5390 ts_tablet_manager.cc:1875] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.278602 5390 ts_tablet_manager.cc:1888] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.278795 5390 log.cc:1196] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/e16034fbf5ce43cebd5d1d3a217b79b3
I20240501 14:00:36.279321 5390 ts_tablet_manager.cc:1909] T e16034fbf5ce43cebd5d1d3a217b79b3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:36.282802 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet e16034fbf5ce43cebd5d1d3a217b79b3 (table rebalance_test_table_6 [id=e65a0d1b93d54439b841febbf2c0dbc7]) successfully deleted
I20240501 14:00:36.376964 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:36.397799 5295 raft_consensus.cc:1261] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:36.398468 4901 raft_consensus.cc:1261] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:36.400203 5562 consensus_queue.cc:1035] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.402379 5562 consensus_queue.cc:1035] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.424634 2516 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:00:36.450254 2003 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet dba319e59aa84691a3e4ccbb4c365870 (table rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee]). Expected version 1 got 0
I20240501 14:00:36.454560 5593 mvcc.cc:205] Tried to move back new op lower bound from 7022887061079642112 to 7022887060289085440. Current Snapshot: MvccSnapshot[applied={T|T < 7022887061079642112}]
I20240501 14:00:36.454690 5487 mvcc.cc:205] Tried to move back new op lower bound from 7022887061079642112 to 7022887060289085440. Current Snapshot: MvccSnapshot[applied={T|T < 7022887061079642112}]
I20240501 14:00:36.462950 2388 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 14:00:36.469375 5600 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:36.471132 5600 tablet.cc:1620] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:36.482904 5601 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:36.483969 2003 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet dba319e59aa84691a3e4ccbb4c365870 (table rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee]). Expected version 1 got 0
I20240501 14:00:36.484186 5601 tablet.cc:1620] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:36.485862 5275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:36.488910 5144 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:36.493839 5013 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:36.499461 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee] alter complete (version 1)
I20240501 14:00:36.520673 2127 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:36.532461 2003 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet dba319e59aa84691a3e4ccbb4c365870 (table rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee]). Expected version 1 got 0
I20240501 14:00:36.536155 2256 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:36.546880 5607 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:36.548796 5607 tablet.cc:1620] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:36.548732 2003 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet dba319e59aa84691a3e4ccbb4c365870 (table rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee]). Expected version 1 got 0
I20240501 14:00:36.586553 4881 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:00:36.600409 5409 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.618981 5561 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.624444 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:36.634295 5561 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.643975 5600 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:36.644512 5600 tablet.cc:1620] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:36.644579 5601 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:36.645001 5601 tablet.cc:1620] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:36.649071 5607 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:36.649281 5464 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.649663 5607 tablet.cc:1620] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:36.658053 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=6243ae1d068d4a97b1c2482059a2eeee] alter complete (version 2)
I20240501 14:00:36.685518 5408 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.699784 5408 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:36.732367 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:00:36.732594 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:00:36.735986 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 1124d6713d8344dd831e20f40b9d95fb
I20240501 14:00:36.736469 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 6ed5ecc71ee94722adfd4d639a2cc270
I20240501 14:00:36.736706 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 1124d6713d8344dd831e20f40b9d95fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:36.736842 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b
I20240501 14:00:36.736842 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 1124d6713d8344dd831e20f40b9d95fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:36.736992 5389 tablet_replica.cc:330] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:36.737128 5508 tablet_replica.cc:330] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:36.737233 5389 raft_consensus.cc:2229] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.737282 5508 raft_consensus.cc:2229] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:36.737488 5389 raft_consensus.cc:2258] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.737823 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 1124d6713d8344dd831e20f40b9d95fb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:36.738119 5390 tablet_replica.cc:330] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:36.738196 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed5ecc71ee94722adfd4d639a2cc270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:36.738250 5389 ts_tablet_manager.cc:1875] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.738315 5390 raft_consensus.cc:2229] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.738417 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed5ecc71ee94722adfd4d639a2cc270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:36.738544 5508 raft_consensus.cc:2258] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.738607 5390 raft_consensus.cc:2258] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.738910 5392 tablet_replica.cc:330] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:36.739104 5392 raft_consensus.cc:2229] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:36.739253 5390 ts_tablet_manager.cc:1875] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.739274 5508 ts_tablet_manager.cc:1875] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.739418 5392 raft_consensus.cc:2258] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.740073 5392 ts_tablet_manager.cc:1875] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.738683 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:36.743237 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:36.743539 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed5ecc71ee94722adfd4d639a2cc270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:36.743777 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:36.744828 5389 ts_tablet_manager.cc:1888] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.745000 5389 log.cc:1196] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/1124d6713d8344dd831e20f40b9d95fb
I20240501 14:00:36.745288 5508 ts_tablet_manager.cc:1888] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.745390 5389 ts_tablet_manager.cc:1909] T 1124d6713d8344dd831e20f40b9d95fb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:36.745429 5508 log.cc:1196] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/1124d6713d8344dd831e20f40b9d95fb
I20240501 14:00:36.745811 5508 ts_tablet_manager.cc:1909] T 1124d6713d8344dd831e20f40b9d95fb P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:36.746387 5389 tablet_replica.cc:330] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:36.746455 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 1124d6713d8344dd831e20f40b9d95fb (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.746917 5508 tablet_replica.cc:330] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:36.746981 5389 raft_consensus.cc:2229] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.747107 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 1124d6713d8344dd831e20f40b9d95fb (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.747210 5508 raft_consensus.cc:2229] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.747247 5389 raft_consensus.cc:2258] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.747759 5508 raft_consensus.cc:2258] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.747907 5389 ts_tablet_manager.cc:1875] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.748530 5508 ts_tablet_manager.cc:1875] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.750178 5392 ts_tablet_manager.cc:1888] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.750384 5392 log.cc:1196] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/6ed5ecc71ee94722adfd4d639a2cc270
I20240501 14:00:36.750612 5390 ts_tablet_manager.cc:1888] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.750784 5390 log.cc:1196] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/1124d6713d8344dd831e20f40b9d95fb
I20240501 14:00:36.750862 5392 ts_tablet_manager.cc:1909] T 6ed5ecc71ee94722adfd4d639a2cc270 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:36.751253 5390 ts_tablet_manager.cc:1909] T 1124d6713d8344dd831e20f40b9d95fb P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:36.752018 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 6ed5ecc71ee94722adfd4d639a2cc270 (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.752148 5390 tablet_replica.cc:330] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:36.752369 5390 raft_consensus.cc:2229] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.752499 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 1124d6713d8344dd831e20f40b9d95fb (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.752684 5390 raft_consensus.cc:2258] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.752856 5392 tablet_replica.cc:330] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:36.753082 5392 raft_consensus.cc:2229] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:36.753463 5392 raft_consensus.cc:2258] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.753492 5390 ts_tablet_manager.cc:1875] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.754143 5392 ts_tablet_manager.cc:1875] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.755432 5389 ts_tablet_manager.cc:1888] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.756759 5389 log.cc:1196] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/6ed5ecc71ee94722adfd4d639a2cc270
I20240501 14:00:36.757288 5389 ts_tablet_manager.cc:1909] T 6ed5ecc71ee94722adfd4d639a2cc270 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:36.758332 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 6ed5ecc71ee94722adfd4d639a2cc270 (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.760624 5392 ts_tablet_manager.cc:1888] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.760767 5392 log.cc:1196] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/a21d64c0e7fb4852b4cdf9bec0eb6a6b
I20240501 14:00:36.761188 5392 ts_tablet_manager.cc:1909] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:36.764772 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.766016 5390 ts_tablet_manager.cc:1888] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.766184 5390 log.cc:1196] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/a21d64c0e7fb4852b4cdf9bec0eb6a6b
I20240501 14:00:36.766407 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:36.766814 5390 ts_tablet_manager.cc:1909] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:36.767876 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.771416 5508 ts_tablet_manager.cc:1888] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.773705 5508 log.cc:1196] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/a21d64c0e7fb4852b4cdf9bec0eb6a6b
I20240501 14:00:36.774160 5508 ts_tablet_manager.cc:1909] T a21d64c0e7fb4852b4cdf9bec0eb6a6b P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:36.775121 5508 tablet_replica.cc:330] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:36.775008 5600 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:36.775413 5508 raft_consensus.cc:2229] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.775668 5508 raft_consensus.cc:2258] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.776335 5601 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:36.780468 5607 tablet.cc:1722] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:36.781383 5508 ts_tablet_manager.cc:1875] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.788272 5508 ts_tablet_manager.cc:1888] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:36.788524 5508 log.cc:1196] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/6ed5ecc71ee94722adfd4d639a2cc270
I20240501 14:00:36.788913 5508 ts_tablet_manager.cc:1909] T 6ed5ecc71ee94722adfd4d639a2cc270 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:36.789214 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet a21d64c0e7fb4852b4cdf9bec0eb6a6b (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.789965 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 6ed5ecc71ee94722adfd4d639a2cc270 (table rebalance_test_table_8 [id=0ee38338f8ca4451ad7bf090d75cb738]) successfully deleted
I20240501 14:00:36.791091 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=6243ae1d068d4a97b1c2482059a2eeee] alter complete (version 3)
I20240501 14:00:36.833138 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:36.842921 5275 tablet_service.cc:1467] Processing CreateTablet for tablet eb1fa60c64034b7d8c7b4879823b0d83 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.843143 5144 tablet_service.cc:1467] Processing CreateTablet for tablet eb1fa60c64034b7d8c7b4879823b0d83 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.843473 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1fa60c64034b7d8c7b4879823b0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.843564 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1fa60c64034b7d8c7b4879823b0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.844393 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 9935f0f87637467091ff85989503af28 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.844642 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 9935f0f87637467091ff85989503af28 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.844784 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935f0f87637467091ff85989503af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.845039 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935f0f87637467091ff85989503af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.847903 5013 tablet_service.cc:1467] Processing CreateTablet for tablet a56d8b76fc244a86b73a741a937bd1c6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.848335 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56d8b76fc244a86b73a741a937bd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.852209 5274 tablet_service.cc:1467] Processing CreateTablet for tablet a56d8b76fc244a86b73a741a937bd1c6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.852730 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56d8b76fc244a86b73a741a937bd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.854437 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 9935f0f87637467091ff85989503af28 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.854877 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935f0f87637467091ff85989503af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.855260 2516 tablet_service.cc:1467] Processing CreateTablet for tablet eb1fa60c64034b7d8c7b4879823b0d83 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.855652 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1fa60c64034b7d8c7b4879823b0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.858966 2127 tablet_service.cc:1467] Processing CreateTablet for tablet a56d8b76fc244a86b73a741a937bd1c6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:36.859417 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56d8b76fc244a86b73a741a937bd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:36.861107 5638 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:36.863631 5638 tablet_bootstrap.cc:654] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.873786 5638 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:36.874087 5638 ts_tablet_manager.cc:1367] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240501 14:00:36.875056 5638 raft_consensus.cc:348] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:36.875820 5638 raft_consensus.cc:374] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.875982 5638 raft_consensus.cc:729] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.876339 5638 consensus_queue.cc:260] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:36.880759 5638 ts_tablet_manager.cc:1398] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:36.880895 5641 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:36.883814 5639 tablet_bootstrap.cc:492] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:36.884846 5643 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:36.885464 5639 tablet_bootstrap.cc:654] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.885887 5641 tablet_bootstrap.cc:654] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.886730 5643 tablet_bootstrap.cc:654] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.892844 5637 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:36.893821 2002 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:36.894084 2002 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:36.896076 5637 tablet_bootstrap.cc:654] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.899304 2002 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet dba319e59aa84691a3e4ccbb4c365870
I20240501 14:00:36.900463 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet dba319e59aa84691a3e4ccbb4c365870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:36.900527 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet dba319e59aa84691a3e4ccbb4c365870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:36.905200 5641 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:36.905812 5641 ts_tablet_manager.cc:1367] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.005s
I20240501 14:00:36.908000 5641 raft_consensus.cc:348] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.900768 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet dba319e59aa84691a3e4ccbb4c365870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:36 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:36.914793 5649 tablet_replica.cc:330] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:36.915010 5649 raft_consensus.cc:2229] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:36.915376 5639 tablet_bootstrap.cc:492] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:36.915617 5639 ts_tablet_manager.cc:1367] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.001s
I20240501 14:00:36.916473 5639 raft_consensus.cc:348] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:36.916791 5641 raft_consensus.cc:374] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.917155 5641 raft_consensus.cc:729] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.917721 5641 consensus_queue.cc:260] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.917296 5650 tablet_replica.cc:330] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:36.918226 5650 raft_consensus.cc:2229] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.918545 5650 raft_consensus.cc:2258] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.919018 5644 tablet_bootstrap.cc:492] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:36.919549 5650 ts_tablet_manager.cc:1875] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.922281 5641 ts_tablet_manager.cc:1398] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 14:00:36.917940 5649 raft_consensus.cc:2258] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.922782 5639 raft_consensus.cc:374] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.922905 5639 raft_consensus.cc:729] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.924057 5640 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:36.924456 5649 ts_tablet_manager.cc:1875] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.924582 5639 consensus_queue.cc:260] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:36.926898 5644 tablet_bootstrap.cc:654] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.927911 5643 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:36.928187 5643 ts_tablet_manager.cc:1367] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.000s
I20240501 14:00:36.929311 5640 tablet_bootstrap.cc:654] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.929244 5643 raft_consensus.cc:348] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.929466 5643 raft_consensus.cc:374] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.929571 5643 raft_consensus.cc:729] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.929805 5643 consensus_queue.cc:260] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.928002 5651 tablet_replica.cc:330] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:36.931360 5639 ts_tablet_manager.cc:1398] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 14:00:36.931665 5651 raft_consensus.cc:2229] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:36.931974 5651 raft_consensus.cc:2258] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:36.933332 5643 ts_tablet_manager.cc:1398] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:36.935849 5521 raft_consensus.cc:482] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:36.936007 5521 raft_consensus.cc:504] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.936704 5521 leader_election.cc:288] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:36.937101 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1fa60c64034b7d8c7b4879823b0d83" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:36.937731 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1fa60c64034b7d8c7b4879823b0d83" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:36.937949 5164 raft_consensus.cc:2454] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
W20240501 14:00:36.938658 2473 leader_election.cc:341] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:36.938787 5637 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:36.939030 2471 leader_election.cc:302] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:36.939090 5637 ts_tablet_manager.cc:1367] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.002s
I20240501 14:00:36.939543 5521 raft_consensus.cc:2781] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:36.939658 5521 raft_consensus.cc:482] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:36.939738 5521 raft_consensus.cc:3037] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.939924 5637 raft_consensus.cc:348] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.940099 5637 raft_consensus.cc:374] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.940121 5651 ts_tablet_manager.cc:1875] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:36.940183 5637 raft_consensus.cc:729] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.940444 5637 consensus_queue.cc:260] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.940959 5637 ts_tablet_manager.cc:1398] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:36.941251 5637 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:36.943084 5637 tablet_bootstrap.cc:654] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.942463 5521 raft_consensus.cc:504] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.944767 5521 leader_election.cc:288] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:36.945290 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1fa60c64034b7d8c7b4879823b0d83" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:36.945513 5164 raft_consensus.cc:3037] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.948423 5164 raft_consensus.cc:2454] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:36.948927 2471 leader_election.cc:302] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:36.949231 5521 raft_consensus.cc:2781] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:36.949366 5521 raft_consensus.cc:686] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:36.949713 5521 consensus_queue.cc:237] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:36.952912 5640 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:36.952980 5649 ts_tablet_manager.cc:1888] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:36.953127 5649 log.cc:1196] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/dba319e59aa84691a3e4ccbb4c365870
I20240501 14:00:36.953210 5640 ts_tablet_manager.cc:1367] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20240501 14:00:36.953866 2003 catalog_manager.cc:5484] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:36.954344 5650 ts_tablet_manager.cc:1888] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:36.954779 5649 ts_tablet_manager.cc:1909] T dba319e59aa84691a3e4ccbb4c365870 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:36.954803 5640 raft_consensus.cc:348] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:36.955662 5640 raft_consensus.cc:374] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.955812 5640 raft_consensus.cc:729] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.956220 5644 tablet_bootstrap.cc:492] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:36.956418 5644 ts_tablet_manager.cc:1367] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.000s
I20240501 14:00:36.956439 5650 log.cc:1196] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/dba319e59aa84691a3e4ccbb4c365870
I20240501 14:00:36.957228 5644 raft_consensus.cc:348] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:36.957494 5644 raft_consensus.cc:374] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.957515 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet dba319e59aa84691a3e4ccbb4c365870 (table rebalancer_dynamic_table_new_name [id=6243ae1d068d4a97b1c2482059a2eeee]) successfully deleted
I20240501 14:00:36.957674 5644 raft_consensus.cc:729] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.957985 5650 ts_tablet_manager.cc:1909] T dba319e59aa84691a3e4ccbb4c365870 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:36.956146 5640 consensus_queue.cc:260] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:36.959264 5644 consensus_queue.cc:260] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:36.960361 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet dba319e59aa84691a3e4ccbb4c365870 (table rebalancer_dynamic_table_new_name [id=6243ae1d068d4a97b1c2482059a2eeee]) successfully deleted
I20240501 14:00:36.960640 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1fa60c64034b7d8c7b4879823b0d83" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:36.960848 5296 raft_consensus.cc:3037] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:36.963407 5644 ts_tablet_manager.cc:1398] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:36.965505 5640 ts_tablet_manager.cc:1398] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 14:00:36.967356 5296 raft_consensus.cc:2454] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:36.970535 5637 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:36.971093 5637 ts_tablet_manager.cc:1367] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20240501 14:00:36.972062 5637 raft_consensus.cc:348] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:36.972373 5637 raft_consensus.cc:374] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.972553 5637 raft_consensus.cc:729] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.972839 5637 consensus_queue.cc:260] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:36.973503 5637 ts_tablet_manager.cc:1398] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:36.973856 5637 tablet_bootstrap.cc:492] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:36.975564 5637 tablet_bootstrap.cc:654] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:36.979753 5651 ts_tablet_manager.cc:1888] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:36.979920 5651 log.cc:1196] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/dba319e59aa84691a3e4ccbb4c365870
I20240501 14:00:36.980495 5651 ts_tablet_manager.cc:1909] T dba319e59aa84691a3e4ccbb4c365870 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:36.981895 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet dba319e59aa84691a3e4ccbb4c365870 (table rebalancer_dynamic_table_new_name [id=6243ae1d068d4a97b1c2482059a2eeee]) successfully deleted
I20240501 14:00:36.983762 5637 tablet_bootstrap.cc:492] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:36.983908 5637 ts_tablet_manager.cc:1367] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 14:00:36.984622 5637 raft_consensus.cc:348] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:36.984869 5637 raft_consensus.cc:374] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:36.984994 5637 raft_consensus.cc:729] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:36.985311 5637 consensus_queue.cc:260] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:36.985941 5637 ts_tablet_manager.cc:1398] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:37.001821 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:37.012733 5274 tablet_service.cc:1467] Processing CreateTablet for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d9efbc73f7f344c88afcae3df2fa7128]), partition=
I20240501 14:00:37.012750 5144 tablet_service.cc:1467] Processing CreateTablet for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d9efbc73f7f344c88afcae3df2fa7128]), partition=
I20240501 14:00:37.013149 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.013186 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.018904 5013 tablet_service.cc:1467] Processing CreateTablet for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d9efbc73f7f344c88afcae3df2fa7128]), partition=
I20240501 14:00:37.019322 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.025050 5637 tablet_bootstrap.cc:492] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:37.027150 5637 tablet_bootstrap.cc:654] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.031731 5639 tablet_bootstrap.cc:492] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:37.032470 5637 tablet_bootstrap.cc:492] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:37.032639 5637 ts_tablet_manager.cc:1367] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240501 14:00:37.033285 5639 tablet_bootstrap.cc:654] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.033424 5637 raft_consensus.cc:348] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.033649 5637 raft_consensus.cc:374] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.033710 5637 raft_consensus.cc:729] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.034281 5641 tablet_bootstrap.cc:492] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:37.033948 5637 consensus_queue.cc:260] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.034827 5637 ts_tablet_manager.cc:1398] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:37.036165 5641 tablet_bootstrap.cc:654] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.041575 5639 tablet_bootstrap.cc:492] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:37.041889 5641 tablet_bootstrap.cc:492] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:37.042052 5641 ts_tablet_manager.cc:1367] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:00:37.042369 5639 ts_tablet_manager.cc:1367] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240501 14:00:37.042798 5641 raft_consensus.cc:348] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.043001 5641 raft_consensus.cc:374] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.043049 5641 raft_consensus.cc:729] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.043174 5639 raft_consensus.cc:348] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.043259 5641 consensus_queue.cc:260] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.043397 5639 raft_consensus.cc:374] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.043466 5639 raft_consensus.cc:729] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.043696 5639 consensus_queue.cc:260] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.044062 5641 ts_tablet_manager.cc:1398] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:37.044201 5639 ts_tablet_manager.cc:1398] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:37.075812 5616 raft_consensus.cc:482] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:37.076009 5616 raft_consensus.cc:504] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.076766 5616 leader_election.cc:288] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:37.077816 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2f106711ba4eb1a9c2dbfbe1256f6c" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:37.078091 5033 raft_consensus.cc:2454] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:37.078822 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2f106711ba4eb1a9c2dbfbe1256f6c" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:37.079068 5296 raft_consensus.cc:2454] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:37.079552 5099 leader_election.cc:302] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:37.080128 5616 raft_consensus.cc:2781] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:37.080273 5616 raft_consensus.cc:482] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:37.080365 5616 raft_consensus.cc:3037] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.083101 5616 raft_consensus.cc:504] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.085458 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2f106711ba4eb1a9c2dbfbe1256f6c" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:37.085711 5296 raft_consensus.cc:3037] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.085939 5616 leader_election.cc:288] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:37.086830 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb2f106711ba4eb1a9c2dbfbe1256f6c" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:37.087041 5033 raft_consensus.cc:3037] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.088937 5296 raft_consensus.cc:2454] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:37.089428 5099 leader_election.cc:302] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:37.089672 5033 raft_consensus.cc:2454] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:37.089798 5616 raft_consensus.cc:2781] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:37.089916 5616 raft_consensus.cc:686] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:37.090241 5616 consensus_queue.cc:237] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.095494 2004 catalog_manager.cc:5484] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:37.104578 5648 raft_consensus.cc:482] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:37.104795 5648 raft_consensus.cc:504] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:37.105612 5648 leader_election.cc:288] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:37.141315 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935f0f87637467091ff85989503af28" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:37.141570 5296 raft_consensus.cc:2454] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:37.142256 2213 leader_election.cc:302] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:37.142592 5648 raft_consensus.cc:2781] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:37.142741 5648 raft_consensus.cc:482] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:37.142828 5648 raft_consensus.cc:3037] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.145298 5648 raft_consensus.cc:504] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:37.145917 5648 leader_election.cc:288] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:37.146567 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935f0f87637467091ff85989503af28" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:37.146806 5296 raft_consensus.cc:3037] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.149577 5296 raft_consensus.cc:2454] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:37.150108 2213 leader_election.cc:302] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:37.150687 5648 raft_consensus.cc:2781] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:37.150889 5648 raft_consensus.cc:686] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:37.151214 5648 consensus_queue.cc:237] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:37.154989 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935f0f87637467091ff85989503af28" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:37.155231 4900 raft_consensus.cc:2454] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:37.155045 1999 catalog_manager.cc:5484] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:37.156176 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9935f0f87637467091ff85989503af28" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:37.156400 4901 raft_consensus.cc:3037] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.159245 4901 raft_consensus.cc:2454] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:37.192672 4760 raft_consensus.cc:482] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:37.192865 4760 raft_consensus.cc:504] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:37.193574 4760 leader_election.cc:288] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:37.194712 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d8b76fc244a86b73a741a937bd1c6" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:37.194972 5296 raft_consensus.cc:2454] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:37.195410 2082 leader_election.cc:302] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:37.195719 4760 raft_consensus.cc:2781] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:37.195829 4760 raft_consensus.cc:482] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:37.195927 4760 raft_consensus.cc:3037] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.199257 4760 raft_consensus.cc:504] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:37.199995 4760 leader_election.cc:288] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:37.200491 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d8b76fc244a86b73a741a937bd1c6" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:37.200680 5296 raft_consensus.cc:3037] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.201169 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d8b76fc244a86b73a741a937bd1c6" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:37.201280 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56d8b76fc244a86b73a741a937bd1c6" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:37.201442 5033 raft_consensus.cc:2454] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:37.202813 5296 raft_consensus.cc:2454] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:37.203174 2082 leader_election.cc:302] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:37.203649 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:37.203445 4760 raft_consensus.cc:2781] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:37.204324 4760 raft_consensus.cc:686] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:37.204548 4760 consensus_queue.cc:237] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:37.208388 1999 catalog_manager.cc:5484] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:37.212108 5296 raft_consensus.cc:1261] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:37.212862 5617 consensus_queue.cc:1035] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.213682 5034 raft_consensus.cc:1261] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:37.214514 5617 consensus_queue.cc:1035] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.217847 2003 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (table rebalancer_dynamic_table [id=d9efbc73f7f344c88afcae3df2fa7128]). Expected version 1 got 0
I20240501 14:00:37.218171 5618 mvcc.cc:205] Tried to move back new op lower bound from 7022887064410247168 to 7022887063925796864. Current Snapshot: MvccSnapshot[applied={T|T < 7022887064410247168}]
I20240501 14:00:37.218863 5601 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:37.219380 5601 tablet.cc:1620] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:37.218863 5700 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:37.220666 5600 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:37.221153 5600 tablet.cc:1620] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:37.222040 1999 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (table rebalancer_dynamic_table [id=d9efbc73f7f344c88afcae3df2fa7128]). Expected version 1 got 0
I20240501 14:00:37.223075 5700 tablet.cc:1620] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:37.230016 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=d9efbc73f7f344c88afcae3df2fa7128] alter complete (version 1)
I20240501 14:00:37.240392 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:00:37.240649 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:00:37.245666 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 623b8a9d77534554ae9fa11f75547a61
I20240501 14:00:37.246624 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 623b8a9d77534554ae9fa11f75547a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:37.246898 5508 tablet_replica.cc:330] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:37.247148 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet 623b8a9d77534554ae9fa11f75547a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:37.247164 5508 raft_consensus.cc:2229] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.247426 5392 tablet_replica.cc:330] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:37.247438 5508 raft_consensus.cc:2258] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.247632 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 96a4b31bcf104c09a3e24d72a3ed9326
I20240501 14:00:37.247632 5392 raft_consensus.cc:2229] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:37.247953 5392 raft_consensus.cc:2258] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.248292 5508 ts_tablet_manager.cc:1875] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.248667 5392 ts_tablet_manager.cc:1875] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.248826 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 96a4b31bcf104c09a3e24d72a3ed9326 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:37.249089 5390 tablet_replica.cc:330] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:37.249300 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet dd33692bc8c942628f2e403e38a52e99
I20240501 14:00:37.249302 5390 raft_consensus.cc:2229] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:37.249645 5390 raft_consensus.cc:2258] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.250322 5390 ts_tablet_manager.cc:1875] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.250628 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet 96a4b31bcf104c09a3e24d72a3ed9326 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:37.250643 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet dd33692bc8c942628f2e403e38a52e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:37.250663 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 623b8a9d77534554ae9fa11f75547a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:37.250892 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 96a4b31bcf104c09a3e24d72a3ed9326 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:37.250933 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet dd33692bc8c942628f2e403e38a52e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:37.251025 5389 tablet_replica.cc:330] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:37.251272 5389 raft_consensus.cc:2229] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.251477 5389 raft_consensus.cc:2258] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.252126 5389 ts_tablet_manager.cc:1875] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.255184 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet dd33692bc8c942628f2e403e38a52e99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:37.257130 5392 ts_tablet_manager.cc:1888] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.259075 5390 ts_tablet_manager.cc:1888] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.259305 5390 log.cc:1196] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/96a4b31bcf104c09a3e24d72a3ed9326
I20240501 14:00:37.259673 5392 log.cc:1196] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/623b8a9d77534554ae9fa11f75547a61
I20240501 14:00:37.259971 5390 ts_tablet_manager.cc:1909] T 96a4b31bcf104c09a3e24d72a3ed9326 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:37.260069 5392 ts_tablet_manager.cc:1909] T 623b8a9d77534554ae9fa11f75547a61 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:37.264487 5392 tablet_replica.cc:330] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:37.266371 5392 raft_consensus.cc:2229] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.267731 5392 raft_consensus.cc:2258] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.269362 5392 ts_tablet_manager.cc:1875] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.271274 5508 ts_tablet_manager.cc:1888] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.271922 5508 log.cc:1196] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/623b8a9d77534554ae9fa11f75547a61
I20240501 14:00:37.273759 5508 ts_tablet_manager.cc:1909] T 623b8a9d77534554ae9fa11f75547a61 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:37.274480 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 96a4b31bcf104c09a3e24d72a3ed9326 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.274914 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 623b8a9d77534554ae9fa11f75547a61 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.276238 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 623b8a9d77534554ae9fa11f75547a61 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.276360 5508 tablet_replica.cc:330] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:37.276841 5508 raft_consensus.cc:2229] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:37.277688 5508 raft_consensus.cc:2258] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.281219 5508 ts_tablet_manager.cc:1875] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.285254 5392 ts_tablet_manager.cc:1888] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.286409 5013 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:37.287128 5274 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:00:37.287288 5392 log.cc:1196] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/96a4b31bcf104c09a3e24d72a3ed9326
I20240501 14:00:37.289503 5392 ts_tablet_manager.cc:1909] T 96a4b31bcf104c09a3e24d72a3ed9326 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:37.290426 5389 ts_tablet_manager.cc:1888] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.291285 5508 ts_tablet_manager.cc:1888] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.291514 5508 log.cc:1196] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd33692bc8c942628f2e403e38a52e99
I20240501 14:00:37.292076 5508 ts_tablet_manager.cc:1909] T dd33692bc8c942628f2e403e38a52e99 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:37.292435 5392 tablet_replica.cc:330] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:37.292666 5392 raft_consensus.cc:2229] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.292906 5392 raft_consensus.cc:2258] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.293311 5508 tablet_replica.cc:330] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:37.293640 5508 raft_consensus.cc:2229] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.293893 5508 raft_consensus.cc:2258] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.294699 5508 ts_tablet_manager.cc:1875] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.294700 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 96a4b31bcf104c09a3e24d72a3ed9326 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.295006 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet dd33692bc8c942628f2e403e38a52e99 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.295527 5392 ts_tablet_manager.cc:1875] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.304963 5389 log.cc:1196] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/623b8a9d77534554ae9fa11f75547a61
I20240501 14:00:37.306098 5389 ts_tablet_manager.cc:1909] T 623b8a9d77534554ae9fa11f75547a61 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:37.307650 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 623b8a9d77534554ae9fa11f75547a61 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.308733 5392 ts_tablet_manager.cc:1888] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.311084 5392 log.cc:1196] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/dd33692bc8c942628f2e403e38a52e99
I20240501 14:00:37.311558 5389 tablet_replica.cc:330] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:37.311807 5392 ts_tablet_manager.cc:1909] T dd33692bc8c942628f2e403e38a52e99 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:37.311946 5508 ts_tablet_manager.cc:1888] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.312196 5508 log.cc:1196] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/96a4b31bcf104c09a3e24d72a3ed9326
I20240501 14:00:37.312626 5508 ts_tablet_manager.cc:1909] T 96a4b31bcf104c09a3e24d72a3ed9326 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:37.312258 5389 raft_consensus.cc:2229] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.313453 2516 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:00:37.313419 5389 raft_consensus.cc:2258] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.313868 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet dd33692bc8c942628f2e403e38a52e99 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.314893 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 96a4b31bcf104c09a3e24d72a3ed9326 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.317577 5389 ts_tablet_manager.cc:1875] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:00:37.321422 2341 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.4:36117, user_credentials={real_user=slave}} blocked reactor thread for 47575.7us
I20240501 14:00:37.335525 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:37.349610 5389 ts_tablet_manager.cc:1888] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.351083 5389 log.cc:1196] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd33692bc8c942628f2e403e38a52e99
I20240501 14:00:37.352934 5700 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:37.353494 5700 tablet.cc:1620] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:37.353798 5389 ts_tablet_manager.cc:1909] T dd33692bc8c942628f2e403e38a52e99 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:37.355240 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet dd33692bc8c942628f2e403e38a52e99 (table rebalance_test_table_10 [id=3ae75dfeb61643d8a2938660ddfa7f71]) successfully deleted
I20240501 14:00:37.356990 5600 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:37.357551 5600 tablet.cc:1620] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:37.362815 2258 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:37.364066 5601 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:37.364593 5601 tablet.cc:1620] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:37.374922 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=d9efbc73f7f344c88afcae3df2fa7128] alter complete (version 2)
I20240501 14:00:37.384572 4881 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:00:37.388916 2341 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:41309, user_credentials={real_user=slave}} blocked reactor thread for 56131.9us
I20240501 14:00:37.398619 2386 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:00:37.407080 5144 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:37.429895 2127 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:00:37.478260 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:37.490458 5700 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:37.491994 5600 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:37.500236 5601 tablet.cc:1722] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:37.502135 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=d9efbc73f7f344c88afcae3df2fa7128] alter complete (version 3)
I20240501 14:00:37.537012 5648 consensus_queue.cc:1035] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.550429 5720 consensus_queue.cc:1035] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.554006 5521 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.563812 5426 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.600139 2538 consensus_queue.cc:237] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:37.603935 2143 raft_consensus.cc:1261] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.604903 5727 consensus_queue.cc:1035] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.606046 2278 raft_consensus.cc:1261] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.606671 5727 consensus_queue.cc:1035] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.610265 5521 raft_consensus.cc:2932] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:37.611574 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:37.611065 2143 raft_consensus.cc:2932] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:37.611797 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240501 14:00:37.612120 2473 consensus_peers.cc:485] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47fda0a6eca040779405bcdb44a15d13. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:37.612890 2278 raft_consensus.cc:2932] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:37.616547 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet eb2f106711ba4eb1a9c2dbfbe1256f6c
I20240501 14:00:37.617380 1997 catalog_manager.cc:5484] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:37.617588 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:37.618202 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:37.618861 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet eb2f106711ba4eb1a9c2dbfbe1256f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:37.655619 2407 consensus_queue.cc:237] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:37.659766 5739 tablet_replica.cc:330] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:37.660396 5738 tablet_replica.cc:330] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:37.660727 5739 raft_consensus.cc:2229] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.661688 5739 raft_consensus.cc:2258] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.661958 5738 raft_consensus.cc:2229] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.662441 5738 raft_consensus.cc:2258] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.663486 5738 ts_tablet_manager.cc:1875] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.666889 5739 ts_tablet_manager.cc:1875] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.666985 5737 tablet_replica.cc:330] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:37.667795 5737 raft_consensus.cc:2229] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:37.669447 2278 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.670241 5737 raft_consensus.cc:2258] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.670181 5713 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.673652 5737 ts_tablet_manager.cc:1875] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.677053 5739 ts_tablet_manager.cc:1888] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:37.677227 5739 log.cc:1196] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/eb2f106711ba4eb1a9c2dbfbe1256f6c
I20240501 14:00:37.679714 2143 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.680624 5738 ts_tablet_manager.cc:1888] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:37.680591 4800 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.680920 5738 log.cc:1196] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/eb2f106711ba4eb1a9c2dbfbe1256f6c
I20240501 14:00:37.681424 5738 ts_tablet_manager.cc:1909] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:37.682741 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (table rebalancer_dynamic_table_new_name [id=d9efbc73f7f344c88afcae3df2fa7128]) successfully deleted
I20240501 14:00:37.683514 5739 ts_tablet_manager.cc:1909] T eb2f106711ba4eb1a9c2dbfbe1256f6c P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:37.684818 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (table rebalancer_dynamic_table_new_name [id=d9efbc73f7f344c88afcae3df2fa7128]) successfully deleted
I20240501 14:00:37.684912 5737 ts_tablet_manager.cc:1888] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:37.685078 5737 log.cc:1196] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/eb2f106711ba4eb1a9c2dbfbe1256f6c
I20240501 14:00:37.685714 5737 ts_tablet_manager.cc:1909] T eb2f106711ba4eb1a9c2dbfbe1256f6c P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:37.692005 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet eb2f106711ba4eb1a9c2dbfbe1256f6c (table rebalancer_dynamic_table_new_name [id=d9efbc73f7f344c88afcae3df2fa7128]) successfully deleted
I20240501 14:00:37.692811 4800 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:37.697144 2278 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:37.699083 2143 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:37.705732 2003 catalog_manager.cc:5484] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
W20240501 14:00:37.706668 2342 consensus_peers.cc:485] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44b7e8e96f4e4cc0bd2c824882ac4d84. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:37.709116 4760 consensus_queue.cc:1035] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.722203 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:37.733819 5034 raft_consensus.cc:3037] T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:37.737658 5446 consensus_queue.cc:1035] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:37.738133 2386 tablet_service.cc:1467] Processing CreateTablet for tablet a934e21a102e42658beca7732818202d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]), partition=
I20240501 14:00:37.738577 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a934e21a102e42658beca7732818202d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.738713 2258 tablet_service.cc:1467] Processing CreateTablet for tablet a934e21a102e42658beca7732818202d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]), partition=
I20240501 14:00:37.739045 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a934e21a102e42658beca7732818202d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.744712 5013 tablet_service.cc:1467] Processing CreateTablet for tablet a934e21a102e42658beca7732818202d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]), partition=
I20240501 14:00:37.744602 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:37.745290 5756 tablet_bootstrap.cc:492] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:37.745664 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a934e21a102e42658beca7732818202d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.746829 5756 tablet_bootstrap.cc:654] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.757709 2006 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:00:37.757997 2006 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:00:37.762823 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 4 replicas of tablet 47fda0a6eca040779405bcdb44a15d13
I20240501 14:00:37.763104 2143 consensus_queue.cc:237] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:37.763487 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 32164bd2be9a49eabc787ab9085db994
I20240501 14:00:37.764336 5014 tablet_service.cc:1514] Processing DeleteTablet for tablet 47fda0a6eca040779405bcdb44a15d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:37.764811 4881 tablet_service.cc:1467] Processing CreateTablet for tablet bf051cecef424db5accfce66623346b5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.765273 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf051cecef424db5accfce66623346b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.765812 4880 tablet_service.cc:1467] Processing CreateTablet for tablet f9b8c92907e843409e37919a1a0e96af (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.766305 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b8c92907e843409e37919a1a0e96af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.766986 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 32164bd2be9a49eabc787ab9085db994 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:37.767697 5144 tablet_service.cc:1467] Processing CreateTablet for tablet bf051cecef424db5accfce66623346b5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.768087 5389 tablet_replica.cc:330] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:37.768337 5389 raft_consensus.cc:2229] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.768486 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9e01283eb491abde0417ffbaff56f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.768595 5389 raft_consensus.cc:2258] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.768857 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9e01283eb491abde0417ffbaff56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.769268 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 47fda0a6eca040779405bcdb44a15d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:37.769930 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf051cecef424db5accfce66623346b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.770937 5143 tablet_service.cc:1467] Processing CreateTablet for tablet f9b8c92907e843409e37919a1a0e96af (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.771366 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b8c92907e843409e37919a1a0e96af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.773721 5761 tablet_replica.cc:330] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:37.774260 5761 raft_consensus.cc:2229] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:37.774569 5389 ts_tablet_manager.cc:1875] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.774686 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 32164bd2be9a49eabc787ab9085db994 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:37.775372 5274 tablet_service.cc:1467] Processing CreateTablet for tablet f9b8c92907e843409e37919a1a0e96af (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.775836 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b8c92907e843409e37919a1a0e96af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.776000 5760 tablet_bootstrap.cc:492] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:37.777561 5761 raft_consensus.cc:2258] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.778805 5761 ts_tablet_manager.cc:1875] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.778928 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9e01283eb491abde0417ffbaff56f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.779173 5760 tablet_bootstrap.cc:654] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.779865 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9e01283eb491abde0417ffbaff56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.783017 5392 tablet_replica.cc:330] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:37.783263 5392 raft_consensus.cc:2229] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:37.783668 5392 raft_consensus.cc:2258] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.784197 5756 tablet_bootstrap.cc:492] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:37.784350 5756 ts_tablet_manager.cc:1367] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.007s
I20240501 14:00:37.784523 5392 ts_tablet_manager.cc:1875] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.785096 5756 raft_consensus.cc:348] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.785303 5756 raft_consensus.cc:374] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.785389 5756 raft_consensus.cc:729] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.785611 5756 consensus_queue.cc:260] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.786134 5756 ts_tablet_manager.cc:1398] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:37.763912 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet e2d270a4918a46c08f699272101cb0fc
W20240501 14:00:37.773274 1992 catalog_manager.cc:4797] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): delete failed for tablet 47fda0a6eca040779405bcdb44a15d13 because the tablet was not found. No further retry: Not found: Tablet not found: 47fda0a6eca040779405bcdb44a15d13
I20240501 14:00:37.787693 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d270a4918a46c08f699272101cb0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:37.788700 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d270a4918a46c08f699272101cb0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:37.788838 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d270a4918a46c08f699272101cb0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:37.790331 5761 ts_tablet_manager.cc:1888] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:37.790490 5761 log.cc:1196] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/47fda0a6eca040779405bcdb44a15d13
I20240501 14:00:37.792558 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 32164bd2be9a49eabc787ab9085db994 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:37.792558 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 47fda0a6eca040779405bcdb44a15d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:37.792879 5508 tablet_replica.cc:330] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:37.793095 5508 raft_consensus.cc:2229] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.793359 5508 raft_consensus.cc:2258] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.793387 5761 ts_tablet_manager.cc:1909] T 47fda0a6eca040779405bcdb44a15d13 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:37.794098 5508 ts_tablet_manager.cc:1875] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.794524 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 47fda0a6eca040779405bcdb44a15d13 (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.794860 5761 tablet_replica.cc:330] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:37.798753 5761 raft_consensus.cc:2229] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:37.799153 5761 raft_consensus.cc:2258] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.800279 5761 ts_tablet_manager.cc:1875] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.803566 5508 ts_tablet_manager.cc:1888] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.803694 5508 log.cc:1196] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/32164bd2be9a49eabc787ab9085db994
I20240501 14:00:37.805835 5508 ts_tablet_manager.cc:1909] T 32164bd2be9a49eabc787ab9085db994 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:37.809079 5764 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:37.810370 5389 ts_tablet_manager.cc:1888] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.810631 5389 log.cc:1196] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/32164bd2be9a49eabc787ab9085db994
I20240501 14:00:37.811602 2275 raft_consensus.cc:1261] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.811734 2538 raft_consensus.cc:1261] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.811775 5389 ts_tablet_manager.cc:1909] T 32164bd2be9a49eabc787ab9085db994 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
W20240501 14:00:37.812389 2082 consensus_peers.cc:485] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6c38a8bcb4a4e07aa424fd074d5e17e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:37.812601 5508 tablet_replica.cc:330] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:37.812919 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 32164bd2be9a49eabc787ab9085db994 (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.813004 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 47fda0a6eca040779405bcdb44a15d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:37 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:37.812656 4760 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.814071 5508 raft_consensus.cc:2229] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.814183 5763 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:37.815006 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 32164bd2be9a49eabc787ab9085db994 (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.815265 5446 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.816063 5763 tablet_bootstrap.cc:654] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.814478 5508 raft_consensus.cc:2258] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.817577 5389 tablet_replica.cc:330] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:37.817785 5764 tablet_bootstrap.cc:654] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.818806 5389 raft_consensus.cc:2229] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.820327 5389 raft_consensus.cc:2258] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.820607 5508 ts_tablet_manager.cc:1875] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.821458 5389 ts_tablet_manager.cc:1875] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.825574 5761 ts_tablet_manager.cc:1888] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.825738 5761 log.cc:1196] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/e2d270a4918a46c08f699272101cb0fc
I20240501 14:00:37.832720 5761 ts_tablet_manager.cc:1909] T e2d270a4918a46c08f699272101cb0fc P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:37.835160 5760 tablet_bootstrap.cc:492] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:37.835381 5760 ts_tablet_manager.cc:1367] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.012s
I20240501 14:00:37.836649 5760 raft_consensus.cc:348] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.837606 5760 raft_consensus.cc:374] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.837754 5760 raft_consensus.cc:729] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.838006 5760 consensus_queue.cc:260] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.836925 5392 ts_tablet_manager.cc:1888] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.840907 5392 log.cc:1196] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/32164bd2be9a49eabc787ab9085db994
I20240501 14:00:37.842229 5508 ts_tablet_manager.cc:1888] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:37.840926 5760 ts_tablet_manager.cc:1398] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:37.841768 5392 ts_tablet_manager.cc:1909] T 32164bd2be9a49eabc787ab9085db994 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:37.844377 5392 tablet_replica.cc:330] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:37.844492 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 32164bd2be9a49eabc787ab9085db994 (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.844584 5392 raft_consensus.cc:2229] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.844810 5392 raft_consensus.cc:2258] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.845554 5392 ts_tablet_manager.cc:1875] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.846789 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet e2d270a4918a46c08f699272101cb0fc (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.847724 5508 log.cc:1196] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/47fda0a6eca040779405bcdb44a15d13
I20240501 14:00:37.848307 4760 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:37.849129 2275 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:37.851019 5765 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:37.856205 1997 catalog_manager.cc:5484] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:37.856917 5765 tablet_bootstrap.cc:654] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.862309 5392 ts_tablet_manager.cc:1888] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.862514 5392 log.cc:1196] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/e2d270a4918a46c08f699272101cb0fc
I20240501 14:00:37.862939 5392 ts_tablet_manager.cc:1909] T e2d270a4918a46c08f699272101cb0fc P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:37.863489 5764 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:37.863687 5764 ts_tablet_manager.cc:1367] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.009s
I20240501 14:00:37.864439 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet e2d270a4918a46c08f699272101cb0fc (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.864617 5764 raft_consensus.cc:348] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.864857 5764 raft_consensus.cc:374] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.864948 5764 raft_consensus.cc:729] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.865250 5764 consensus_queue.cc:260] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.865816 5764 ts_tablet_manager.cc:1398] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:37.866165 5764 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:37.875918 5763 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:37.876230 5763 ts_tablet_manager.cc:1367] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.006s
I20240501 14:00:37.877190 5763 raft_consensus.cc:348] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.877589 5763 raft_consensus.cc:374] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.877732 5763 raft_consensus.cc:729] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.878065 5763 consensus_queue.cc:260] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.854488 5014 tablet_service.cc:1467] Processing CreateTablet for tablet bf051cecef424db5accfce66623346b5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.878623 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf051cecef424db5accfce66623346b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.879233 5763 ts_tablet_manager.cc:1398] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:37.880391 5763 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:37.883419 5763 tablet_bootstrap.cc:654] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.883783 5764 tablet_bootstrap.cc:654] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.885063 2538 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:37.886782 5765 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:37.886996 5765 ts_tablet_manager.cc:1367] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.014s
I20240501 14:00:37.887908 5765 raft_consensus.cc:348] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.888108 5765 raft_consensus.cc:374] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.888182 5765 raft_consensus.cc:729] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.888437 5765 consensus_queue.cc:260] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.889276 5765 ts_tablet_manager.cc:1398] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:37.895838 5763 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:37.896392 5763 ts_tablet_manager.cc:1367] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.003s
I20240501 14:00:37.897132 5763 raft_consensus.cc:348] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.897334 5763 raft_consensus.cc:374] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.897411 5763 raft_consensus.cc:729] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.897658 5763 consensus_queue.cc:260] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.898295 5763 ts_tablet_manager.cc:1398] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:37.899334 5764 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:37.899562 5764 ts_tablet_manager.cc:1367] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.001s
I20240501 14:00:37.902792 5764 raft_consensus.cc:348] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.903162 5764 raft_consensus.cc:374] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.903291 5764 raft_consensus.cc:729] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.903600 5764 consensus_queue.cc:260] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.906579 5764 ts_tablet_manager.cc:1398] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:37.907029 5763 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:37.909152 5763 tablet_bootstrap.cc:654] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.910946 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9e01283eb491abde0417ffbaff56f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:37.911374 5764 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:37.911513 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9e01283eb491abde0417ffbaff56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:37.913429 5764 tablet_bootstrap.cc:654] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.913944 5508 ts_tablet_manager.cc:1909] T 47fda0a6eca040779405bcdb44a15d13 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:37.915244 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 47fda0a6eca040779405bcdb44a15d13 (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.915766 5389 ts_tablet_manager.cc:1888] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:37.915927 5389 log.cc:1196] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/e2d270a4918a46c08f699272101cb0fc
I20240501 14:00:37.916361 5389 ts_tablet_manager.cc:1909] T e2d270a4918a46c08f699272101cb0fc P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:37.916373 5763 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:37.916528 5763 ts_tablet_manager.cc:1367] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240501 14:00:37.917387 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet e2d270a4918a46c08f699272101cb0fc (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.917353 5763 raft_consensus.cc:348] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.917551 5763 raft_consensus.cc:374] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.917631 5763 raft_consensus.cc:729] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.917891 5763 consensus_queue.cc:260] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
W20240501 14:00:37.911113 5757 log.cc:924] Time spent T a56d8b76fc244a86b73a741a937bd1c6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20240501 14:00:37.921694 5763 ts_tablet_manager.cc:1398] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:00:37.923173 5774 tablet_bootstrap.cc:492] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:37.925594 5764 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:37.925779 5764 ts_tablet_manager.cc:1367] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20240501 14:00:37.926605 5764 raft_consensus.cc:348] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.926822 5764 raft_consensus.cc:374] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.926939 5764 raft_consensus.cc:729] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.927184 5764 consensus_queue.cc:260] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.927755 5764 ts_tablet_manager.cc:1398] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:37.928129 5389 tablet_replica.cc:330] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:37.928668 5389 raft_consensus.cc:2229] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:37.929208 5389 raft_consensus.cc:2258] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:37.930565 5389 ts_tablet_manager.cc:1875] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:37.931788 5774 tablet_bootstrap.cc:654] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.948751 5389 ts_tablet_manager.cc:1888] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:37.951859 5389 log.cc:1196] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/47fda0a6eca040779405bcdb44a15d13
I20240501 14:00:37.952405 5389 ts_tablet_manager.cc:1909] T 47fda0a6eca040779405bcdb44a15d13 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:37.954928 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 47fda0a6eca040779405bcdb44a15d13 (table rebalance_test_table_12 [id=0c5b27d0993d4334bdefc10c3cb3309a]) successfully deleted
I20240501 14:00:37.958034 5774 tablet_bootstrap.cc:492] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:37.958241 5774 ts_tablet_manager.cc:1367] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.006s
I20240501 14:00:37.959082 5774 raft_consensus.cc:348] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.959311 5774 raft_consensus.cc:374] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.959400 5774 raft_consensus.cc:729] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.959623 5774 consensus_queue.cc:260] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.963567 5774 ts_tablet_manager.cc:1398] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:37.963924 5774 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:37.966068 5774 tablet_bootstrap.cc:654] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.968351 5774 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:37.968593 5774 ts_tablet_manager.cc:1367] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:37.969385 5774 raft_consensus.cc:348] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.969674 5774 raft_consensus.cc:374] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.969775 5774 raft_consensus.cc:729] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.970082 5774 consensus_queue.cc:260] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:37.970784 5774 ts_tablet_manager.cc:1398] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:37.971133 5774 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:37.972748 5774 tablet_bootstrap.cc:654] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:37.974897 5774 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:37.975051 5774 ts_tablet_manager.cc:1367] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:37.975865 5774 raft_consensus.cc:348] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.976137 5774 raft_consensus.cc:374] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:37.976231 5774 raft_consensus.cc:729] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:37.976548 5774 consensus_queue.cc:260] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:37.977106 5774 ts_tablet_manager.cc:1398] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:37.989526 2407 consensus_queue.cc:237] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:37.992496 2538 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.992589 2146 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:37.993196 5713 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.993607 4799 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:37.997443 5713 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:37.999362 2146 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:38.004437 2006 catalog_manager.cc:5484] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:38.008872 2538 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
W20240501 14:00:38.016496 2341 consensus_peers.cc:485] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fac8d3842d994279b355f9a18a82f901. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.017882 5365 raft_consensus.cc:482] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.018059 5365 raft_consensus.cc:504] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:38.018884 5365 leader_election.cc:288] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:38.020699 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b8c92907e843409e37919a1a0e96af" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:38.020973 5164 raft_consensus.cc:2454] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:38.021575 5228 leader_election.cc:302] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:38.022176 5365 raft_consensus.cc:2781] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.022308 5365 raft_consensus.cc:482] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.022437 5365 raft_consensus.cc:3037] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.025658 5365 raft_consensus.cc:504] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:38.026643 5365 leader_election.cc:288] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:38.027254 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b8c92907e843409e37919a1a0e96af" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:38.027456 5164 raft_consensus.cc:3037] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.029497 5164 raft_consensus.cc:2454] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:38.029984 5228 leader_election.cc:302] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:38.030289 5365 raft_consensus.cc:2781] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.034787 5365 raft_consensus.cc:686] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:38.035248 5365 consensus_queue.cc:237] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:38.036116 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b8c92907e843409e37919a1a0e96af" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:38.036697 4900 raft_consensus.cc:3037] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.039129 5772 raft_consensus.cc:482] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.039288 5772 raft_consensus.cc:504] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:38.039398 4900 raft_consensus.cc:2454] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:38.036648 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b8c92907e843409e37919a1a0e96af" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:38.039865 4901 raft_consensus.cc:2362] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1. Re-sending same reply.
I20240501 14:00:38.040236 5772 leader_election.cc:288] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:38.040918 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf051cecef424db5accfce66623346b5" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:38.041170 4901 raft_consensus.cc:2454] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:38.041674 4968 leader_election.cc:302] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:38.042027 5772 raft_consensus.cc:2781] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.042165 5772 raft_consensus.cc:482] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.042251 5772 raft_consensus.cc:3037] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.045289 5772 raft_consensus.cc:504] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:38.045912 5772 leader_election.cc:288] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:38.047344 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf051cecef424db5accfce66623346b5" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:38.047592 4901 raft_consensus.cc:3037] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.047808 2006 catalog_manager.cc:5484] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:38.052814 4901 raft_consensus.cc:2454] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:38.053263 4968 leader_election.cc:302] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:38.053689 5772 raft_consensus.cc:2781] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.053901 5772 raft_consensus.cc:686] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:38.054311 5772 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:38.057210 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf051cecef424db5accfce66623346b5" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:38.057338 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf051cecef424db5accfce66623346b5" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:38.057472 5164 raft_consensus.cc:2454] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:38.057315 2146 consensus_queue.cc:237] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:38.060171 2538 raft_consensus.cc:1261] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:38.060740 2082 consensus_peers.cc:485] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28f67dedd5f44301b00d3e9c08330198. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.060765 5770 consensus_queue.cc:1035] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.060966 2278 raft_consensus.cc:1261] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.060768 2006 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:38.061661 5770 consensus_queue.cc:1035] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.063813 5770 raft_consensus.cc:2932] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:38.065773 2278 raft_consensus.cc:2932] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:38.066996 2538 raft_consensus.cc:2932] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:38.070454 2006 catalog_manager.cc:5484] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:38.088092 2278 consensus_queue.cc:237] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:38.090984 2407 raft_consensus.cc:1261] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.091312 2538 raft_consensus.cc:1261] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.091624 5648 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:38.092010 2212 consensus_peers.cc:485] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3738397b81b46868e09071be6f82334. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.092100 5507 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.095947 5648 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:38.096712 2407 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:38.101485 5364 raft_consensus.cc:482] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.101668 5364 raft_consensus.cc:504] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:38.102061 1997 catalog_manager.cc:5484] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:38.102561 5364 leader_election.cc:288] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:38.106210 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9e01283eb491abde0417ffbaff56f" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:38.106463 5163 raft_consensus.cc:2454] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:38.107153 4834 leader_election.cc:302] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:38.107600 5364 raft_consensus.cc:2781] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.107700 5364 raft_consensus.cc:482] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.107796 5364 raft_consensus.cc:3037] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.111832 5364 raft_consensus.cc:504] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:38.113160 5364 leader_election.cc:288] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:38.113293 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9e01283eb491abde0417ffbaff56f" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:38.113580 5163 raft_consensus.cc:3037] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.116559 5163 raft_consensus.cc:2454] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:38.117061 4834 leader_election.cc:302] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:38.117405 5364 raft_consensus.cc:2781] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.116328 2538 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:38.123148 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9e01283eb491abde0417ffbaff56f" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:38.123435 5034 raft_consensus.cc:2454] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:38.123800 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9e01283eb491abde0417ffbaff56f" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:38.123986 5033 raft_consensus.cc:3037] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.126868 5033 raft_consensus.cc:2454] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:38.133783 4799 raft_consensus.cc:482] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.134035 4799 raft_consensus.cc:504] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:38.134384 5364 raft_consensus.cc:686] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:38.134691 5364 consensus_queue.cc:237] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:38.135054 4799 leader_election.cc:288] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:38.135531 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a934e21a102e42658beca7732818202d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:38.135756 2278 raft_consensus.cc:2454] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:38.137794 2341 leader_election.cc:302] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:38.138136 4799 raft_consensus.cc:2781] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.138247 4799 raft_consensus.cc:482] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.138991 4799 raft_consensus.cc:3037] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.142129 4799 raft_consensus.cc:504] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:38.143163 4799 leader_election.cc:288] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:38.143304 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a934e21a102e42658beca7732818202d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:38.143532 2278 raft_consensus.cc:3037] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.145393 2278 raft_consensus.cc:2454] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:38.146109 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a934e21a102e42658beca7732818202d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:38.146071 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a934e21a102e42658beca7732818202d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:38.146340 5034 raft_consensus.cc:3037] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.148654 5034 raft_consensus.cc:2454] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:38.149824 2343 leader_election.cc:302] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:38.150142 4799 raft_consensus.cc:2781] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.150305 4799 raft_consensus.cc:686] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:38.150614 4799 consensus_queue.cc:237] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:38.159891 2407 consensus_queue.cc:237] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:38.162057 1997 catalog_manager.cc:5484] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:38.163002 2006 catalog_manager.cc:5484] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:38.173272 2538 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.173883 2278 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.174746 4799 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.175343 5713 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:38.182686 2343 consensus_peers.cc:485] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e048df08cf4f7eaacb78d6592d79b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.194834 5713 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:38.195492 2538 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:38.197765 2275 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:38.202113 2006 catalog_manager.cc:5484] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:38.212529 5830 ts_tablet_manager.cc:902] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:38.228008 5830 tablet_copy_client.cc:308] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:38.235810 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:49081)
I20240501 14:00:38.236097 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:49081: session id = f8c60f2bbac541dcba4424ab726f9ad9-44b7e8e96f4e4cc0bd2c824882ac4d84
I20240501 14:00:38.239102 2538 consensus_queue.cc:237] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:38.239252 2428 tablet_copy_source_session.cc:205] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:38.242079 2275 raft_consensus.cc:1261] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.242231 2146 raft_consensus.cc:1261] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.242449 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:00:38.243130 2471 consensus_peers.cc:485] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98e027e2a36541eda1a67c0a3ab48d65. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.243505 5727 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.243791 5521 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.247188 5521 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:38.247830 2146 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:38.254760 2006 catalog_manager.cc:5484] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:38.256013 5830 tablet_copy_client.cc:783] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:38.256440 5830 tablet_copy_client.cc:647] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:38.260197 5830 tablet_copy_client.cc:523] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:38.263530 2275 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:38.264771 5830 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:38.284946 2407 consensus_queue.cc:237] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:38.287899 2275 raft_consensus.cc:1261] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.288211 2146 raft_consensus.cc:1261] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.288836 4799 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.289156 4800 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:38.289681 2343 consensus_peers.cc:485] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b1f151470f24501857f38461d5b70c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.292670 2006 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:00:38.292905 2006 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:00:38.293872 4800 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:38.296950 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 3682f3837f6e40fbbe32e47d948d1305
I20240501 14:00:38.297488 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet bc35182173f2418e91a852adc391f5a5
I20240501 14:00:38.297880 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 8756f19007eb4704a80213dd469874a7
I20240501 14:00:38.298154 2278 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:38.299577 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 3682f3837f6e40fbbe32e47d948d1305 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:38.299701 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 8756f19007eb4704a80213dd469874a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:38.299939 5389 tablet_replica.cc:330] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:38.300140 5389 raft_consensus.cc:2229] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.300390 5389 raft_consensus.cc:2258] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.300966 5389 ts_tablet_manager.cc:1875] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.301612 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet bc35182173f2418e91a852adc391f5a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:38.302106 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet bc35182173f2418e91a852adc391f5a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:38.302657 5761 tablet_replica.cc:330] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:38.302840 5761 raft_consensus.cc:2229] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.303076 5761 raft_consensus.cc:2258] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.303656 5761 ts_tablet_manager.cc:1875] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.305860 5389 ts_tablet_manager.cc:1888] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.305985 5389 log.cc:1196] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/3682f3837f6e40fbbe32e47d948d1305
I20240501 14:00:38.306311 5389 ts_tablet_manager.cc:1909] T 3682f3837f6e40fbbe32e47d948d1305 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:38.307056 2386 tablet_service.cc:1514] Processing DeleteTablet for tablet 3682f3837f6e40fbbe32e47d948d1305 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:38.307335 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet bc35182173f2418e91a852adc391f5a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:38.307534 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 8756f19007eb4704a80213dd469874a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:38.307631 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 8756f19007eb4704a80213dd469874a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:38.307737 5392 tablet_replica.cc:330] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:38.307929 5392 raft_consensus.cc:2229] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.308190 5392 raft_consensus.cc:2258] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.308928 5392 ts_tablet_manager.cc:1875] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.309486 2146 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:38.310722 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 3682f3837f6e40fbbe32e47d948d1305 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:38.311010 5508 tablet_replica.cc:330] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:38.311208 5508 raft_consensus.cc:2229] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.311472 5508 raft_consensus.cc:2258] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.311447 2003 catalog_manager.cc:5484] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:38.312184 5508 ts_tablet_manager.cc:1875] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.319802 5761 ts_tablet_manager.cc:1888] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.320057 5761 log.cc:1196] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/bc35182173f2418e91a852adc391f5a5
I20240501 14:00:38.320586 5761 ts_tablet_manager.cc:1909] T bc35182173f2418e91a852adc391f5a5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:38.321520 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 3682f3837f6e40fbbe32e47d948d1305 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.322955 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet bc35182173f2418e91a852adc391f5a5 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.323369 5389 tablet_replica.cc:330] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:38.323698 5389 raft_consensus.cc:2229] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:38.324093 5389 raft_consensus.cc:2258] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.325102 5389 ts_tablet_manager.cc:1875] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.330219 5392 ts_tablet_manager.cc:1888] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.330319 5830 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:38.330389 5392 log.cc:1196] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/3682f3837f6e40fbbe32e47d948d1305
I20240501 14:00:38.330772 5830 tablet_bootstrap.cc:492] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:38.330806 5392 ts_tablet_manager.cc:1909] T 3682f3837f6e40fbbe32e47d948d1305 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:38.331039 5830 ts_tablet_manager.cc:1367] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.066s user 0.048s sys 0.007s
I20240501 14:00:38.331486 5508 ts_tablet_manager.cc:1888] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.331647 5508 log.cc:1196] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/8756f19007eb4704a80213dd469874a7
I20240501 14:00:38.332070 5508 ts_tablet_manager.cc:1909] T 8756f19007eb4704a80213dd469874a7 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:38.331956 5830 raft_consensus.cc:348] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:38.332252 5830 raft_consensus.cc:729] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:38.332474 5830 consensus_queue.cc:260] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:38.333057 5508 tablet_replica.cc:330] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:38.333289 5508 raft_consensus.cc:2229] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:38.333482 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 8756f19007eb4704a80213dd469874a7 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.333611 5508 raft_consensus.cc:2258] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.333941 5830 ts_tablet_manager.cc:1398] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:38.334816 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-44b7e8e96f4e4cc0bd2c824882ac4d84 received from {username='slave'} at 127.28.17.5:49081
I20240501 14:00:38.334968 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-44b7e8e96f4e4cc0bd2c824882ac4d84 on tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:38.335523 5851 ts_tablet_manager.cc:902] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:38.336347 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 3682f3837f6e40fbbe32e47d948d1305 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.338600 5392 tablet_replica.cc:330] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:38.338886 5392 raft_consensus.cc:2229] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:38.339286 5392 raft_consensus.cc:2258] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.334312 5508 ts_tablet_manager.cc:1875] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.343811 5392 ts_tablet_manager.cc:1875] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.346382 5389 ts_tablet_manager.cc:1888] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.346612 5389 log.cc:1196] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/8756f19007eb4704a80213dd469874a7
I20240501 14:00:38.347075 5389 ts_tablet_manager.cc:1909] T 8756f19007eb4704a80213dd469874a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:38.348248 5389 tablet_replica.cc:330] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:38.348208 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 8756f19007eb4704a80213dd469874a7 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.348474 5389 raft_consensus.cc:2229] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.348775 5389 raft_consensus.cc:2258] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.349537 5389 ts_tablet_manager.cc:1875] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.355914 2006 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:38.356572 5508 ts_tablet_manager.cc:1888] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.356683 5389 ts_tablet_manager.cc:1888] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.356745 5508 log.cc:1196] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/3682f3837f6e40fbbe32e47d948d1305
I20240501 14:00:38.356806 5389 log.cc:1196] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/bc35182173f2418e91a852adc391f5a5
I20240501 14:00:38.357209 5508 ts_tablet_manager.cc:1909] T 3682f3837f6e40fbbe32e47d948d1305 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:38.359326 5389 ts_tablet_manager.cc:1909] T bc35182173f2418e91a852adc391f5a5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:38.359575 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 3682f3837f6e40fbbe32e47d948d1305 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.361377 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet bc35182173f2418e91a852adc391f5a5 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.362834 5392 ts_tablet_manager.cc:1888] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.363873 5392 log.cc:1196] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc35182173f2418e91a852adc391f5a5
I20240501 14:00:38.364431 5392 ts_tablet_manager.cc:1909] T bc35182173f2418e91a852adc391f5a5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:38.367383 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet bc35182173f2418e91a852adc391f5a5 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.369237 2278 raft_consensus.cc:1261] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:38.369256 5034 raft_consensus.cc:1261] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:38.369896 5392 tablet_replica.cc:330] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:38.370204 5392 raft_consensus.cc:2229] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.370492 5392 raft_consensus.cc:2258] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.370108 4799 consensus_queue.cc:1035] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.371443 4800 consensus_queue.cc:1035] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.372577 5392 ts_tablet_manager.cc:1875] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.379971 5851 tablet_copy_client.cc:308] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:00:38.380864 5392 ts_tablet_manager.cc:1888] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.381420 5392 log.cc:1196] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/8756f19007eb4704a80213dd469874a7
I20240501 14:00:38.381992 5392 ts_tablet_manager.cc:1909] T 8756f19007eb4704a80213dd469874a7 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:38.383154 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 8756f19007eb4704a80213dd469874a7 (table rebalance_test_table_14 [id=469f028ef0fc4bb4b259e720495fbfc9]) successfully deleted
I20240501 14:00:38.387997 2006 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]). Expected version 1 got 0
I20240501 14:00:38.389840 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet b6c38a8bcb4a4e07aa424fd074d5e17e from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:50475)
I20240501 14:00:38.390064 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet b6c38a8bcb4a4e07aa424fd074d5e17e from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:50475: session id = 5a4b9a96e9734baa8a03d6056bfa934c-b6c38a8bcb4a4e07aa424fd074d5e17e
I20240501 14:00:38.390156 5866 tablet.cc:1722] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:38.390746 5866 tablet.cc:1620] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:38.389189 5865 tablet.cc:1722] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:38.391701 5865 tablet.cc:1620] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:38.393834 2167 tablet_copy_source_session.cc:205] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:38.396210 5867 ts_tablet_manager.cc:902] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:38.402078 2006 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]). Expected version 1 got 0
I20240501 14:00:38.403949 5851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c38a8bcb4a4e07aa424fd074d5e17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.404394 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e] alter complete (version 1)
I20240501 14:00:38.416303 2538 consensus_queue.cc:237] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:38.419958 2406 raft_consensus.cc:1261] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.420591 2146 raft_consensus.cc:1261] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.421586 5727 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.424618 5727 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 14:00:38.428745 5727 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:38.433756 2143 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:38.440399 2006 catalog_manager.cc:5484] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:38.413115 5851 tablet_copy_client.cc:783] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:38.455994 5365 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240501 14:00:38.463263 5851 tablet_copy_client.cc:647] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:38.469075 5851 tablet_copy_client.cc:523] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:38.479893 5772 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.486305 2407 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:38.490092 5162 raft_consensus.cc:3037] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.493728 5851 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:38.493808 5772 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.497035 2003 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]). Expected version 1 got 0
I20240501 14:00:38.511986 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240501 14:00:38.521032 2472 consensus_peers.cc:485] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87a79bfca7cd4ffcbb0e4448320d0576. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.526309 5865 tablet.cc:1722] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:38.526968 5865 tablet.cc:1620] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:38.527503 5866 tablet.cc:1722] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:38.528012 5866 tablet.cc:1620] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:38.529289 5889 ts_tablet_manager.cc:902] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:38.536379 5830 ts_tablet_manager.cc:902] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:38.536862 5364 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.546022 2006 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e] alter complete (version 2)
I20240501 14:00:38.555757 5885 tablet.cc:1722] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:38.556558 5885 tablet.cc:1620] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:38.558887 5796 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.559474 2006 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]). Expected version 2 got 0
I20240501 14:00:38.564833 5889 tablet_copy_client.cc:308] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:00:38.566401 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet 28f67dedd5f44301b00d3e9c08330198 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:50703)
I20240501 14:00:38.566699 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet 28f67dedd5f44301b00d3e9c08330198 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:50703: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-28f67dedd5f44301b00d3e9c08330198
I20240501 14:00:38.569566 2167 tablet_copy_source_session.cc:205] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:38.570592 5364 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:38.586131 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f67dedd5f44301b00d3e9c08330198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.592128 5867 tablet_copy_client.cc:308] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:38.592756 5885 tablet.cc:1722] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:38.593197 2003 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table [id=95d8a18f01744ad8b66ed5b8f0ec430e]). Expected version 2 got 1
I20240501 14:00:38.593266 5885 tablet.cc:1620] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:38.601217 5889 tablet_copy_client.cc:783] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:38.601709 5889 tablet_copy_client.cc:647] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:38.603479 2276 consensus_queue.cc:237] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:38.606012 5889 tablet_copy_client.cc:523] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:00:38.613256 5229 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.5:40059, user_credentials={real_user=slave}} blocked reactor thread for 54599.7us
I20240501 14:00:38.617053 5889 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:38.619872 2538 raft_consensus.cc:1261] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.620539 5507 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:38.627254 2213 consensus_peers.cc:485] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85842504df3e4571b1a9750809711db2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:38.629796 2143 raft_consensus.cc:1261] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.630513 5507 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.642712 5898 ts_tablet_manager.cc:902] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:38.651661 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:38.662652 5865 tablet.cc:1722] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:38.667034 5830 tablet_copy_client.cc:308] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:00:38.667089 5866 tablet.cc:1722] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:38.667737 5885 tablet.cc:1722] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:38.668498 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet b3738397b81b46868e09071be6f82334 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:52743)
I20240501 14:00:38.668689 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet b3738397b81b46868e09071be6f82334 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:52743: session id = f8c60f2bbac541dcba4424ab726f9ad9-b3738397b81b46868e09071be6f82334
I20240501 14:00:38.670377 5851 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:38.682916 2298 tablet_copy_source_session.cc:205] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:38.683441 4901 raft_consensus.cc:1203] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:38.686528 5898 tablet_copy_client.cc:308] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:38.689639 5851 tablet_bootstrap.cc:492] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:00:38.690143 5851 ts_tablet_manager.cc:1367] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.197s user 0.053s sys 0.012s
I20240501 14:00:38.690282 2536 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:38.691063 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=95d8a18f01744ad8b66ed5b8f0ec430e] alter complete (version 3)
I20240501 14:00:38.692183 5851 raft_consensus.cc:348] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:38.692608 5851 raft_consensus.cc:729] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:00:38.692915 5851 consensus_queue.cc:260] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:38.694125 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:38.695537 5851 ts_tablet_manager.cc:1398] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:38.700425 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3738397b81b46868e09071be6f82334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.706166 2387 tablet_service.cc:1467] Processing CreateTablet for tablet bcf596d2da8849ed9b236efea09f32d9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.706720 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf596d2da8849ed9b236efea09f32d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.707441 2256 tablet_service.cc:1467] Processing CreateTablet for tablet bcf596d2da8849ed9b236efea09f32d9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.707437 4879 tablet_service.cc:1467] Processing CreateTablet for tablet bcf596d2da8849ed9b236efea09f32d9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.707841 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf596d2da8849ed9b236efea09f32d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.707852 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf596d2da8849ed9b236efea09f32d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.710943 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 83392c08bf314fcd98833335c247ccb1 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.711227 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 83392c08bf314fcd98833335c247ccb1 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.711380 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83392c08bf314fcd98833335c247ccb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.711640 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83392c08bf314fcd98833335c247ccb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.714591 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 83392c08bf314fcd98833335c247ccb1 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.715107 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83392c08bf314fcd98833335c247ccb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.715157 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 13d8ec926b7a46568470e824763569e6 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.715554 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d8ec926b7a46568470e824763569e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.717633 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 13d8ec926b7a46568470e824763569e6 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.718041 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d8ec926b7a46568470e824763569e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.721033 5918 tablet_bootstrap.cc:492] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:38.723171 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 13d8ec926b7a46568470e824763569e6 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:38.723649 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d8ec926b7a46568470e824763569e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.729529 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet fac8d3842d994279b355f9a18a82f901 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:46539)
I20240501 14:00:38.729725 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet fac8d3842d994279b355f9a18a82f901 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:46539: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-fac8d3842d994279b355f9a18a82f901
I20240501 14:00:38.729683 1997 catalog_manager.cc:5484] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:38.688750 5912 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:38.732633 2428 tablet_copy_source_session.cc:205] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:38.736088 5867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac8d3842d994279b355f9a18a82f901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.738806 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-b6c38a8bcb4a4e07aa424fd074d5e17e received from {username='slave'} at 127.28.17.8:50475
I20240501 14:00:38.738967 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-b6c38a8bcb4a4e07aa424fd074d5e17e on tablet b6c38a8bcb4a4e07aa424fd074d5e17e with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:38.743778 5918 tablet_bootstrap.cc:654] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.747850 5867 tablet_copy_client.cc:783] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:38.750432 2146 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:38.754592 5921 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:38.748373 5867 tablet_copy_client.cc:647] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:38.760586 5921 tablet_bootstrap.cc:654] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.766889 5867 tablet_copy_client.cc:523] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:38.767472 5830 tablet_copy_client.cc:783] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:38.768251 5830 tablet_copy_client.cc:647] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:38.773399 5923 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:38.775992 5830 tablet_copy_client.cc:523] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:38.775974 5922 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:38.780208 5830 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:38.783962 5922 tablet_bootstrap.cc:654] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.785113 5923 tablet_bootstrap.cc:654] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.787070 5918 tablet_bootstrap.cc:492] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:38.787266 5918 ts_tablet_manager.cc:1367] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.003s
I20240501 14:00:38.788137 5918 raft_consensus.cc:348] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:38.788359 5918 raft_consensus.cc:374] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.788444 5918 raft_consensus.cc:729] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.788719 5918 consensus_queue.cc:260] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:38.789570 5920 tablet_bootstrap.cc:492] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:38.792711 5918 ts_tablet_manager.cc:1398] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:38.793329 5919 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:38.794718 1997 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:38.795010 1997 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:38.795743 5918 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:38.797902 5918 tablet_bootstrap.cc:654] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.798409 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet f1e048df08cf4f7eaacb78d6592d79b1 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:53689)
I20240501 14:00:38.798605 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet f1e048df08cf4f7eaacb78d6592d79b1 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:53689: session id = 5a4b9a96e9734baa8a03d6056bfa934c-f1e048df08cf4f7eaacb78d6592d79b1
I20240501 14:00:38.800271 2006 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:00:38.800560 2006 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:00:38.801626 2428 tablet_copy_source_session.cc:205] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:38.807053 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 4 replicas of tablet 28f67dedd5f44301b00d3e9c08330198
I20240501 14:00:38.807209 5898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e048df08cf4f7eaacb78d6592d79b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.808018 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 48b053e242914219aa46965f405b10c4
I20240501 14:00:38.808497 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f67dedd5f44301b00d3e9c08330198 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:38.808811 5508 tablet_replica.cc:330] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:38.808943 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f67dedd5f44301b00d3e9c08330198 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:38.809023 5508 raft_consensus.cc:2229] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:38.809221 5761 tablet_replica.cc:330] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:38.809448 5761 raft_consensus.cc:2229] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.809454 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 6ef9539679954eb284005573c96789bf
I20240501 14:00:38.809525 5508 raft_consensus.cc:2258] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.809690 5761 raft_consensus.cc:2258] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.810091 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b053e242914219aa46965f405b10c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:38.810379 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef9539679954eb284005573c96789bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:38.810688 5392 tablet_replica.cc:330] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:38.810855 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f67dedd5f44301b00d3e9c08330198 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:38.810900 5392 raft_consensus.cc:2229] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.811182 5392 raft_consensus.cc:2258] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.811180 5761 ts_tablet_manager.cc:1875] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.811924 5392 ts_tablet_manager.cc:1875] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.812698 5508 ts_tablet_manager.cc:1875] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.813486 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef9539679954eb284005573c96789bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:38.811903 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b053e242914219aa46965f405b10c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:38.819617 1997 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet a934e21a102e42658beca7732818202d
I20240501 14:00:38.821270 5508 ts_tablet_manager.cc:1888] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:38.821411 5508 log.cc:1196] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/28f67dedd5f44301b00d3e9c08330198
I20240501 14:00:38.821535 5012 tablet_service.cc:1514] Processing DeleteTablet for tablet a934e21a102e42658beca7732818202d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:38.821764 5508 ts_tablet_manager.cc:1909] T 28f67dedd5f44301b00d3e9c08330198 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:38.821827 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet a934e21a102e42658beca7732818202d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:38.822242 5508 tablet_replica.cc:330] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:38.822371 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 28f67dedd5f44301b00d3e9c08330198 (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.822476 5508 raft_consensus.cc:2229] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.822822 5508 raft_consensus.cc:2258] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.823186 5392 ts_tablet_manager.cc:1888] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.823381 5392 log.cc:1196] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/6ef9539679954eb284005573c96789bf
I20240501 14:00:38.823765 5508 ts_tablet_manager.cc:1875] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.823813 5392 ts_tablet_manager.cc:1909] T 6ef9539679954eb284005573c96789bf P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:38.823871 5898 tablet_copy_client.cc:783] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:38.824398 5898 tablet_copy_client.cc:647] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:38.824801 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 6ef9539679954eb284005573c96789bf (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.825202 5392 tablet_replica.cc:330] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:38.825541 5392 raft_consensus.cc:2229] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:38.825989 5392 raft_consensus.cc:2258] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.827081 5392 ts_tablet_manager.cc:1875] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.827147 5761 ts_tablet_manager.cc:1888] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:38.827893 5761 log.cc:1196] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/28f67dedd5f44301b00d3e9c08330198
I20240501 14:00:38.828532 5761 ts_tablet_manager.cc:1909] T 28f67dedd5f44301b00d3e9c08330198 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:38.830179 5898 tablet_copy_client.cc:523] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:38.834057 5508 ts_tablet_manager.cc:1888] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.834635 5508 log.cc:1196] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/48b053e242914219aa46965f405b10c4
I20240501 14:00:38.835340 5508 ts_tablet_manager.cc:1909] T 48b053e242914219aa46965f405b10c4 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:38.836760 5898 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:38.837309 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 48b053e242914219aa46965f405b10c4 (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.837560 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 28f67dedd5f44301b00d3e9c08330198 (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.837613 5761 tablet_replica.cc:330] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:38.837810 5761 raft_consensus.cc:2229] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.838114 5920 tablet_bootstrap.cc:654] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.833596 5923 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:38.840396 5923 ts_tablet_manager.cc:1367] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240501 14:00:38.841363 5923 raft_consensus.cc:348] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.810592 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b053e242914219aa46965f405b10c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:38.845845 5389 tablet_replica.cc:330] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:38.846118 5389 raft_consensus.cc:2229] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.846408 5389 raft_consensus.cc:2258] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.847214 5389 ts_tablet_manager.cc:1875] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.811703 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f67dedd5f44301b00d3e9c08330198 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:38.812062 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef9539679954eb284005573c96789bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:38.821048 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet a934e21a102e42658beca7732818202d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:38.850673 5867 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:38.851766 5933 ts_tablet_manager.cc:902] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:38.852936 5919 tablet_bootstrap.cc:654] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.856173 5921 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:38.856171 5389 ts_tablet_manager.cc:1888] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.856349 5389 log.cc:1196] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/48b053e242914219aa46965f405b10c4
I20240501 14:00:38.856599 5921 ts_tablet_manager.cc:1367] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.102s user 0.022s sys 0.025s
I20240501 14:00:38.856869 5389 ts_tablet_manager.cc:1909] T 48b053e242914219aa46965f405b10c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:38.857498 5921 raft_consensus.cc:348] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:38.857801 5921 raft_consensus.cc:374] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.857939 5921 raft_consensus.cc:729] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.858253 5921 consensus_queue.cc:260] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:38.858418 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 48b053e242914219aa46965f405b10c4 (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.858903 5389 tablet_replica.cc:330] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:38.859042 5921 ts_tablet_manager.cc:1398] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:38.859230 5389 raft_consensus.cc:2229] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.859565 5389 raft_consensus.cc:2258] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.868212 5923 raft_consensus.cc:374] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.868589 5923 raft_consensus.cc:729] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.869196 5923 consensus_queue.cc:260] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
W20240501 14:00:38.870436 1992 catalog_manager.cc:4810] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): delete failed for tablet 28f67dedd5f44301b00d3e9c08330198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f67dedd5f44301b00d3e9c08330198 already in progress: copying tablet
I20240501 14:00:38.871340 5389 ts_tablet_manager.cc:1875] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.872407 5922 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:38.872716 5922 ts_tablet_manager.cc:1367] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.099s user 0.006s sys 0.000s
I20240501 14:00:38.873980 5933 tablet_copy_client.cc:308] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:38.874703 5938 ts_tablet_manager.cc:902] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:38.878078 2406 consensus_queue.cc:237] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:38.883093 5923 ts_tablet_manager.cc:1398] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240501 14:00:38.884832 5920 tablet_bootstrap.cc:492] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:38.885077 5920 ts_tablet_manager.cc:1367] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.096s user 0.014s sys 0.010s
I20240501 14:00:38.886108 5920 raft_consensus.cc:348] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:38.840417 5761 raft_consensus.cc:2258] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.886430 5920 raft_consensus.cc:374] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.886679 5920 raft_consensus.cc:729] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.887362 5761 ts_tablet_manager.cc:1875] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.888070 5918 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:38.889266 5932 tablet_replica.cc:330] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:38.889474 5932 raft_consensus.cc:2229] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:38.889770 5932 raft_consensus.cc:2258] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.889776 5920 consensus_queue.cc:260] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:38.891172 5922 raft_consensus.cc:348] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.887779 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 98e027e2a36541eda1a67c0a3ab48d65 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:35785)
I20240501 14:00:38.892072 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 98e027e2a36541eda1a67c0a3ab48d65 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:35785: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-98e027e2a36541eda1a67c0a3ab48d65
I20240501 14:00:38.897459 2558 tablet_copy_source_session.cc:205] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:38.899415 5922 raft_consensus.cc:374] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.899850 5922 raft_consensus.cc:729] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.900481 5922 consensus_queue.cc:260] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.901680 5920 ts_tablet_manager.cc:1398] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240501 14:00:38.902205 5919 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:38.902424 5919 ts_tablet_manager.cc:1367] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.000s
I20240501 14:00:38.903344 5919 raft_consensus.cc:348] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.903787 5919 raft_consensus.cc:374] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.903916 5919 raft_consensus.cc:729] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.904240 5919 consensus_queue.cc:260] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.906044 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e027e2a36541eda1a67c0a3ab48d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.907290 5919 ts_tablet_manager.cc:1398] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:38.907696 5932 ts_tablet_manager.cc:1875] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.908149 5919 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:38.910758 5918 ts_tablet_manager.cc:1367] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.001s
I20240501 14:00:38.910859 5922 ts_tablet_manager.cc:1398] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20240501 14:00:38.914429 5922 tablet_bootstrap.cc:492] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:38.918108 5919 tablet_bootstrap.cc:654] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.919443 5392 ts_tablet_manager.cc:1888] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.919565 5392 log.cc:1196] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/48b053e242914219aa46965f405b10c4
I20240501 14:00:38.919927 5392 ts_tablet_manager.cc:1909] T 48b053e242914219aa46965f405b10c4 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:38.920749 5392 tablet_replica.cc:330] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:38.913187 5933 tablet_copy_client.cc:783] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:38.921916 5933 tablet_copy_client.cc:647] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:38.923177 5918 raft_consensus.cc:348] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:38.923583 5392 raft_consensus.cc:2229] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:38.924057 5392 raft_consensus.cc:2258] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:38.924326 5933 tablet_copy_client.cc:523] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:38.924629 5918 raft_consensus.cc:374] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:38.924680 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 48b053e242914219aa46965f405b10c4 (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.925654 5392 ts_tablet_manager.cc:1875] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:38.926613 2006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:38.929268 5918 raft_consensus.cc:729] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:38.932073 5933 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:38.932713 2276 raft_consensus.cc:1261] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.933638 4799 consensus_queue.cc:1035] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.934023 2146 raft_consensus.cc:1261] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:38.934736 4799 consensus_queue.cc:1035] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:38.937639 5918 consensus_queue.cc:260] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:38.938236 5274 tablet_service.cc:1467] Processing CreateTablet for tablet c32a8d5414fb4364a63004c6c6eafe1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56]), partition=
I20240501 14:00:38.938406 2516 tablet_service.cc:1467] Processing CreateTablet for tablet c32a8d5414fb4364a63004c6c6eafe1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56]), partition=
I20240501 14:00:38.938597 2387 tablet_service.cc:1467] Processing CreateTablet for tablet c32a8d5414fb4364a63004c6c6eafe1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56]), partition=
I20240501 14:00:38.938679 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32a8d5414fb4364a63004c6c6eafe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.938833 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32a8d5414fb4364a63004c6c6eafe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.938926 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32a8d5414fb4364a63004c6c6eafe1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:38.945808 5761 ts_tablet_manager.cc:1888] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:38.946729 5922 tablet_bootstrap.cc:654] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.946900 5944 ts_tablet_manager.cc:902] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:38.947886 5761 log.cc:1196] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/6ef9539679954eb284005573c96789bf
I20240501 14:00:38.948426 5761 ts_tablet_manager.cc:1909] T 6ef9539679954eb284005573c96789bf P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:38.946102 5923 tablet_bootstrap.cc:492] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:38.952545 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 6ef9539679954eb284005573c96789bf (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.953480 5617 raft_consensus.cc:482] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:38.953656 5617 raft_consensus.cc:504] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.954066 5923 tablet_bootstrap.cc:654] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.954586 5617 leader_election.cc:288] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:38.955006 5918 ts_tablet_manager.cc:1398] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20240501 14:00:38.957557 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:38.957805 2538 raft_consensus.cc:2454] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:38.960022 5097 leader_election.cc:302] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:38.961088 5617 raft_consensus.cc:2781] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:38.961227 5617 raft_consensus.cc:482] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:38.961324 5617 raft_consensus.cc:3037] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.970970 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:38.971381 4901 raft_consensus.cc:2454] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:38.970425 5617 raft_consensus.cc:504] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.973928 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:38.974187 2538 raft_consensus.cc:3037] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.975080 5389 ts_tablet_manager.cc:1888] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:38.975219 5389 log.cc:1196] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/28f67dedd5f44301b00d3e9c08330198
I20240501 14:00:38.975633 5389 ts_tablet_manager.cc:1909] T 28f67dedd5f44301b00d3e9c08330198 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:38.977109 2538 raft_consensus.cc:2454] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:38.977826 5921 tablet_bootstrap.cc:492] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:38.977768 5097 leader_election.cc:302] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:38.979243 5617 leader_election.cc:288] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:38.979457 5617 raft_consensus.cc:2781] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:38.979524 5921 tablet_bootstrap.cc:654] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.979578 5617 raft_consensus.cc:686] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:38.980206 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:38.980958 4901 raft_consensus.cc:3037] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:38.981523 5617 consensus_queue.cc:237] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:38.985366 5392 ts_tablet_manager.cc:1888] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:38.986403 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 28f67dedd5f44301b00d3e9c08330198 (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:38.987290 5389 tablet_replica.cc:330] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:38.985461 5920 tablet_bootstrap.cc:492] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:38.990379 5920 tablet_bootstrap.cc:654] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:38.991888 4901 raft_consensus.cc:2454] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:38.995252 5392 log.cc:1196] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/a934e21a102e42658beca7732818202d
I20240501 14:00:38.995678 5389 raft_consensus.cc:2229] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:39.000391 5392 ts_tablet_manager.cc:1909] T a934e21a102e42658beca7732818202d P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:39.001647 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table_new_name [id=95d8a18f01744ad8b66ed5b8f0ec430e]) successfully deleted
W20240501 14:00:38.997560 2211 consensus_peers.cc:485] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ef9539679954eb284005573c96789bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:39.003104 5389 raft_consensus.cc:2258] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.004092 5389 ts_tablet_manager.cc:1875] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.015620 5919 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:39.015924 5919 ts_tablet_manager.cc:1367] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.108s user 0.000s sys 0.003s
I20240501 14:00:39.016778 5919 raft_consensus.cc:348] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:39.017099 5919 raft_consensus.cc:374] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.017232 5919 raft_consensus.cc:729] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.017527 5919 consensus_queue.cc:260] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:39.028280 4799 raft_consensus.cc:2932] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:39.029372 5920 tablet_bootstrap.cc:492] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:39.029572 5920 ts_tablet_manager.cc:1367] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.010s
I20240501 14:00:39.032478 2143 raft_consensus.cc:2932] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
W20240501 14:00:39.044910 5097 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.17.4:36117, user_credentials={real_user=slave}} blocked reactor thread for 67108.8us
I20240501 14:00:39.046383 5932 ts_tablet_manager.cc:1888] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:39.046566 5932 log.cc:1196] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/a934e21a102e42658beca7732818202d
I20240501 14:00:39.046968 5932 ts_tablet_manager.cc:1909] T a934e21a102e42658beca7732818202d P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:39.046787 2006 catalog_manager.cc:5484] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:39.049089 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table_new_name [id=95d8a18f01744ad8b66ed5b8f0ec430e]) successfully deleted
I20240501 14:00:39.050828 5920 raft_consensus.cc:348] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.051717 5920 raft_consensus.cc:374] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.052147 5920 raft_consensus.cc:729] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.052784 5920 consensus_queue.cc:260] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.053159 2278 raft_consensus.cc:2932] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:39.054656 5938 tablet_copy_client.cc:308] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:39.058732 1997 catalog_manager.cc:5484] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:39.062188 5922 tablet_bootstrap.cc:492] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:39.062494 5922 ts_tablet_manager.cc:1367] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.023s
I20240501 14:00:39.063383 5923 tablet_bootstrap.cc:492] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:39.063613 5923 ts_tablet_manager.cc:1367] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.118s user 0.013s sys 0.010s
I20240501 14:00:39.064447 5923 raft_consensus.cc:348] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.065680 5923 raft_consensus.cc:374] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.065982 5923 raft_consensus.cc:729] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.066676 5923 consensus_queue.cc:260] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.066022 5922 raft_consensus.cc:348] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.067886 5922 raft_consensus.cc:374] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.067970 5922 raft_consensus.cc:729] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.071606 5922 consensus_queue.cc:260] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.073298 5922 ts_tablet_manager.cc:1398] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:00:39.078544 5919 ts_tablet_manager.cc:1398] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.061s user 0.000s sys 0.002s
I20240501 14:00:39.078739 5921 tablet_bootstrap.cc:492] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:39.078976 5921 ts_tablet_manager.cc:1367] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.009s
I20240501 14:00:39.079829 5921 raft_consensus.cc:348] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.081297 5902 raft_consensus.cc:482] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:39.081462 5902 raft_consensus.cc:504] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
W20240501 14:00:39.054069 2343 consensus_peers.cc:485] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e58200ed1d53430e97ed77e154dee4a7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:39.082291 5902 leader_election.cc:288] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:39.082594 4799 raft_consensus.cc:482] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:39.082759 4799 raft_consensus.cc:504] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.083169 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:39.083361 2406 raft_consensus.cc:2454] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:39.083424 4799 leader_election.cc:288] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:39.083624 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet 5b1f151470f24501857f38461d5b70c8 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:41439)
I20240501 14:00:39.083766 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet 5b1f151470f24501857f38461d5b70c8 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:41439: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-5b1f151470f24501857f38461d5b70c8
I20240501 14:00:39.080142 5617 raft_consensus.cc:482] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:39.085211 5617 raft_consensus.cc:504] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:39.087399 4834 leader_election.cc:302] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:39.087833 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:39.088054 4901 raft_consensus.cc:2454] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:39.088900 2342 leader_election.cc:302] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:39.088966 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:39.089196 2276 raft_consensus.cc:2454] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:39.089246 4799 raft_consensus.cc:2781] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:39.089349 4799 raft_consensus.cc:482] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:39.089429 4799 raft_consensus.cc:3037] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.089874 2428 tablet_copy_source_session.cc:205] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:39.092586 4799 raft_consensus.cc:504] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.093353 5920 ts_tablet_manager.cc:1398] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.064s user 0.003s sys 0.000s
I20240501 14:00:39.093650 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:39.093879 2276 raft_consensus.cc:3037] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.094056 5816 raft_consensus.cc:2781] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:39.094149 5816 raft_consensus.cc:482] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:39.094240 5816 raft_consensus.cc:3037] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.094455 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d8ec926b7a46568470e824763569e6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:39.094732 2275 raft_consensus.cc:2454] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:39.095206 5097 leader_election.cc:302] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:39.095495 5964 raft_consensus.cc:2781] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:39.095589 5964 raft_consensus.cc:482] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:39.095687 5964 raft_consensus.cc:3037] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.096614 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:39.096925 5816 raft_consensus.cc:504] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.097635 5816 leader_election.cc:288] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:39.099005 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:39.099282 2406 raft_consensus.cc:2379] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8c60f2bbac541dcba4424ab726f9ad9 in current term 1: Already voted for candidate a64f34785a7b44fa8070b0452a4e0ddd in this term.
I20240501 14:00:39.098737 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:39.099716 2276 raft_consensus.cc:2454] T bcf596d2da8849ed9b236efea09f32d9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:39.098325 5964 raft_consensus.cc:504] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:39.100008 4834 leader_election.cc:302] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8c60f2bbac541dcba4424ab726f9ad9; no voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:39.100286 2341 leader_election.cc:302] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:39.100553 4800 raft_consensus.cc:2781] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:39.100545 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf596d2da8849ed9b236efea09f32d9" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:39.100685 4800 raft_consensus.cc:686] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:39.100793 4901 raft_consensus.cc:2379] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a64f34785a7b44fa8070b0452a4e0ddd in current term 1: Already voted for candidate f8c60f2bbac541dcba4424ab726f9ad9 in this term.
I20240501 14:00:39.101016 5902 raft_consensus.cc:2726] T bcf596d2da8849ed9b236efea09f32d9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:00:39.100970 4800 consensus_queue.cc:237] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.101306 4799 leader_election.cc:288] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:39.101972 5964 leader_election.cc:288] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:39.102622 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d8ec926b7a46568470e824763569e6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:39.102890 2278 raft_consensus.cc:3037] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.106087 2278 raft_consensus.cc:2454] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:39.106837 5097 leader_election.cc:302] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:39.107208 5964 raft_consensus.cc:2781] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:39.107393 5964 raft_consensus.cc:686] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:39.107414 5921 raft_consensus.cc:374] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.088302 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d8ec926b7a46568470e824763569e6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:39.107733 5296 raft_consensus.cc:2454] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:39.101614 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d8ec926b7a46568470e824763569e6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:39.107898 5964 consensus_queue.cc:237] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:39.108047 5295 raft_consensus.cc:3037] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.111914 5295 raft_consensus.cc:2454] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:39.107556 5921 raft_consensus.cc:729] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.104146 5617 leader_election.cc:288] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
W20240501 14:00:39.121479 2473 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.8:34221, user_credentials={real_user=slave}} blocked reactor thread for 82090.8us
I20240501 14:00:39.123405 1997 catalog_manager.cc:5484] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:39.126899 2006 catalog_manager.cc:5484] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
W20240501 14:00:39.128782 2471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:41309, user_credentials={real_user=slave}} blocked reactor thread for 89729.2us
I20240501 14:00:39.131762 5295 raft_consensus.cc:1203] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.133023 5923 ts_tablet_manager.cc:1398] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
W20240501 14:00:39.133386 4967 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:41309, user_credentials={real_user=slave}} blocked reactor thread for 66002us
I20240501 14:00:39.137028 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1f151470f24501857f38461d5b70c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.137398 5889 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.137903 5889 tablet_bootstrap.cc:492] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:39.138247 5889 ts_tablet_manager.cc:1367] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.521s user 0.062s sys 0.002s
I20240501 14:00:39.139232 5889 raft_consensus.cc:348] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:39.139580 5889 raft_consensus.cc:729] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:39.139824 5889 consensus_queue.cc:260] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:39.140889 2538 consensus_queue.cc:237] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:00:39.145557 5851 ts_tablet_manager.cc:902] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:39.151698 5389 ts_tablet_manager.cc:1888] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:39.131776 5921 consensus_queue.cc:260] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.156118 5921 ts_tablet_manager.cc:1398] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.077s user 0.002s sys 0.000s
I20240501 14:00:39.156649 5295 raft_consensus.cc:1261] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:39.157290 5976 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:39.157732 5389 log.cc:1196] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/6ef9539679954eb284005573c96789bf
I20240501 14:00:39.158221 5389 ts_tablet_manager.cc:1909] T 6ef9539679954eb284005573c96789bf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:39.160116 5944 tablet_copy_client.cc:308] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:39.171070 5163 raft_consensus.cc:1261] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:00:39.174130 2471 consensus_peers.cc:485] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 052964af500c427e836f0f3121aa91f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:39.177683 5976 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:39.181399 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 6ef9539679954eb284005573c96789bf (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:39.186847 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f67dedd5f44301b00d3e9c08330198 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:39.187181 5389 tablet_replica.cc:330] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
W20240501 14:00:39.187566 1992 catalog_manager.cc:4810] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): delete failed for tablet 28f67dedd5f44301b00d3e9c08330198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f67dedd5f44301b00d3e9c08330198 already in progress: copying tablet
I20240501 14:00:39.188598 5389 raft_consensus.cc:2229] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.188867 5389 raft_consensus.cc:2258] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.189311 5889 ts_tablet_manager.cc:1398] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.051s user 0.002s sys 0.001s
I20240501 14:00:39.194722 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f67dedd5f44301b00d3e9c08330198 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:38 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:39.194591 5727 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:00:39.195029 5932 tablet_replica.cc:330] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:39.195241 5932 raft_consensus.cc:2229] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:00:39.195411 5932 raft_consensus.cc:2258] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:00:39.196203 5932 ts_tablet_manager.cc:1875] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.199834 5389 ts_tablet_manager.cc:1875] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.201185 5898 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.201638 5898 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:00:39.203011 5898 ts_tablet_manager.cc:1367] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.366s user 0.044s sys 0.004s
I20240501 14:00:39.203195 1999 catalog_manager.cc:5484] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:39.203680 5932 ts_tablet_manager.cc:1888] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:39.203927 5932 log.cc:1196] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/28f67dedd5f44301b00d3e9c08330198
I20240501 14:00:39.203837 5898 raft_consensus.cc:348] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:39.204116 5898 raft_consensus.cc:729] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:00:39.204344 5898 consensus_queue.cc:260] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:39.204507 5932 ts_tablet_manager.cc:1909] T 28f67dedd5f44301b00d3e9c08330198 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:39.204926 5898 ts_tablet_manager.cc:1398] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:39.206517 5851 tablet_copy_client.cc:308] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:00:39.206862 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-f1e048df08cf4f7eaacb78d6592d79b1 received from {username='slave'} at 127.28.17.8:53689
I20240501 14:00:39.206986 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-f1e048df08cf4f7eaacb78d6592d79b1 on tablet f1e048df08cf4f7eaacb78d6592d79b1 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:39.206482 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-28f67dedd5f44301b00d3e9c08330198 received from {username='slave'} at 127.28.17.6:50703
I20240501 14:00:39.207429 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-28f67dedd5f44301b00d3e9c08330198 on tablet 28f67dedd5f44301b00d3e9c08330198 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:39.209326 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 28f67dedd5f44301b00d3e9c08330198 (table rebalance_test_table_16 [id=8706214779b84da799aea96f99e9e1eb]) successfully deleted
I20240501 14:00:39.214082 5938 tablet_copy_client.cc:783] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:39.214375 5938 tablet_copy_client.cc:647] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:39.216148 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 85842504df3e4571b1a9750809711db2 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:39671)
I20240501 14:00:39.215557 5163 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:00:39.216357 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 85842504df3e4571b1a9750809711db2 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:39671: session id = 5a4b9a96e9734baa8a03d6056bfa934c-85842504df3e4571b1a9750809711db2
I20240501 14:00:39.219626 2298 tablet_copy_source_session.cc:205] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:39.223565 5851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85842504df3e4571b1a9750809711db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.224359 5295 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:00:39.228268 5938 tablet_copy_client.cc:523] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:39.236979 5389 ts_tablet_manager.cc:1888] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:39.237143 5389 log.cc:1196] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/a934e21a102e42658beca7732818202d
I20240501 14:00:39.237532 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 87a79bfca7cd4ffcbb0e4448320d0576 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:50935)
I20240501 14:00:39.237731 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 87a79bfca7cd4ffcbb0e4448320d0576 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:50935: session id = f8c60f2bbac541dcba4424ab726f9ad9-87a79bfca7cd4ffcbb0e4448320d0576
I20240501 14:00:39.237782 5389 ts_tablet_manager.cc:1909] T a934e21a102e42658beca7732818202d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:39.240906 2558 tablet_copy_source_session.cc:205] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:39.242796 5944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a79bfca7cd4ffcbb0e4448320d0576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.244436 5938 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:39.247390 5994 raft_consensus.cc:482] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:39.248674 5994 raft_consensus.cc:504] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.251109 5994 leader_election.cc:288] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:39.251663 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a8d5414fb4364a63004c6c6eafe1a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:39.251899 2406 raft_consensus.cc:2454] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:39.252926 5228 leader_election.cc:302] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:39.253240 5994 raft_consensus.cc:2781] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:39.253424 5994 raft_consensus.cc:482] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:39.253562 5994 raft_consensus.cc:3037] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.253739 5944 tablet_copy_client.cc:783] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:39.254016 5944 tablet_copy_client.cc:647] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:39.254459 5851 tablet_copy_client.cc:783] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:39.254786 5851 tablet_copy_client.cc:647] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:39.258402 5994 raft_consensus.cc:504] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.258744 5944 tablet_copy_client.cc:523] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:39.258689 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a8d5414fb4364a63004c6c6eafe1a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:39.258946 2538 raft_consensus.cc:2454] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:39.262019 5994 leader_election.cc:288] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:39.264137 5944 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:39.269970 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a8d5414fb4364a63004c6c6eafe1a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:39.270257 2406 raft_consensus.cc:3037] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.261420 5851 tablet_copy_client.cc:523] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:39.272481 5830 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.273921 5830 tablet_bootstrap.cc:492] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:39.274214 5830 ts_tablet_manager.cc:1367] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.494s user 0.045s sys 0.003s
I20240501 14:00:39.275113 5830 raft_consensus.cc:348] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:39.275420 5830 raft_consensus.cc:729] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:39.275671 5830 consensus_queue.cc:260] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:39.275962 5851 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:39.276320 5830 ts_tablet_manager.cc:1398] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:39.277241 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-b3738397b81b46868e09071be6f82334 received from {username='slave'} at 127.28.17.5:52743
I20240501 14:00:39.277406 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-b3738397b81b46868e09071be6f82334 on tablet b3738397b81b46868e09071be6f82334 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:39.280269 2406 raft_consensus.cc:2454] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:39.282153 5228 leader_election.cc:302] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:39.285320 5994 raft_consensus.cc:2781] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:39.287694 5994 raft_consensus.cc:686] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:39.288559 5994 consensus_queue.cc:237] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:39.295573 1999 catalog_manager.cc:5484] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:39.302858 4800 raft_consensus.cc:1050] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:39.304004 4800 consensus_queue.cc:237] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:39.309697 4901 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:39.312553 1999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:00:39.312840 1999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:00:39.313886 2143 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:39.314934 2278 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:39.318167 5713 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:39.319629 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 4 replicas of tablet e58200ed1d53430e97ed77e154dee4a7
I20240501 14:00:39.320662 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet e58200ed1d53430e97ed77e154dee4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:39.320813 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet e58200ed1d53430e97ed77e154dee4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:39.320866 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet e58200ed1d53430e97ed77e154dee4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:39.321143 5392 tablet_replica.cc:330] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:39.321346 5392 raft_consensus.cc:2229] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Raft consensus shutting down.
W20240501 14:00:39.321470 1992 catalog_manager.cc:4797] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): delete failed for tablet e58200ed1d53430e97ed77e154dee4a7 because the tablet was not found. No further retry: Not found: Tablet not found: e58200ed1d53430e97ed77e154dee4a7
I20240501 14:00:39.321785 5392 raft_consensus.cc:2258] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.321697 4799 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:39.322258 4799 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:39.322860 5392 ts_tablet_manager.cc:1875] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.324508 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet a131d188420344e0a76c7b84596194ea
I20240501 14:00:39.324997 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 8afb34c628104a51a0d9c9d4dd649123
I20240501 14:00:39.325109 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet e58200ed1d53430e97ed77e154dee4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:39.325397 5508 tablet_replica.cc:330] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:39.325598 5508 raft_consensus.cc:2229] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.325830 5508 raft_consensus.cc:2258] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.325966 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet a131d188420344e0a76c7b84596194ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:39.326118 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afb34c628104a51a0d9c9d4dd649123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:39.326613 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afb34c628104a51a0d9c9d4dd649123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:39.326891 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet a131d188420344e0a76c7b84596194ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:39.327178 5508 ts_tablet_manager.cc:1875] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.328051 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet a131d188420344e0a76c7b84596194ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:39.328315 5761 tablet_replica.cc:330] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:39.328554 5761 raft_consensus.cc:2229] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:39.328874 5761 raft_consensus.cc:2258] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.329669 5761 ts_tablet_manager.cc:1875] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.330705 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afb34c628104a51a0d9c9d4dd649123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:39 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:39.334349 5508 ts_tablet_manager.cc:1888] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:39.334592 5508 log.cc:1196] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/e58200ed1d53430e97ed77e154dee4a7
I20240501 14:00:39.335090 5508 ts_tablet_manager.cc:1909] T e58200ed1d53430e97ed77e154dee4a7 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:39.338775 5508 tablet_replica.cc:330] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:39.339918 5508 raft_consensus.cc:2229] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:39.340698 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet e58200ed1d53430e97ed77e154dee4a7 (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.341403 5508 raft_consensus.cc:2258] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.342425 5508 ts_tablet_manager.cc:1875] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.356678 5392 ts_tablet_manager.cc:1888] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:39.356861 5392 log.cc:1196] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/e58200ed1d53430e97ed77e154dee4a7
I20240501 14:00:39.357324 5392 ts_tablet_manager.cc:1909] T e58200ed1d53430e97ed77e154dee4a7 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:39.358446 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet e58200ed1d53430e97ed77e154dee4a7 (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.359026 5761 ts_tablet_manager.cc:1888] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:39.359254 5761 log.cc:1196] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/a131d188420344e0a76c7b84596194ea
I20240501 14:00:39.359750 5761 ts_tablet_manager.cc:1909] T a131d188420344e0a76c7b84596194ea P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:39.361493 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a8d5414fb4364a63004c6c6eafe1a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:39.361711 2538 raft_consensus.cc:3037] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:39.362968 5392 tablet_replica.cc:330] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:39.363260 5392 raft_consensus.cc:2229] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.363550 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet a131d188420344e0a76c7b84596194ea (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.363914 5392 raft_consensus.cc:2258] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.364204 5933 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.364650 5933 tablet_bootstrap.cc:492] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:39.364950 5933 ts_tablet_manager.cc:1367] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.433s user 0.058s sys 0.029s
I20240501 14:00:39.365201 2538 raft_consensus.cc:2454] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:39.365316 5508 ts_tablet_manager.cc:1888] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:39.365527 5508 log.cc:1196] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/8afb34c628104a51a0d9c9d4dd649123
I20240501 14:00:39.365756 5392 ts_tablet_manager.cc:1875] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.366022 5508 ts_tablet_manager.cc:1909] T 8afb34c628104a51a0d9c9d4dd649123 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:39.366426 5933 raft_consensus.cc:348] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:39.366730 5933 raft_consensus.cc:729] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:39.366978 5933 consensus_queue.cc:260] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:39.368464 5933 ts_tablet_manager.cc:1398] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 14:00:39.368431 5713 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:39.369428 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-98e027e2a36541eda1a67c0a3ab48d65 received from {username='slave'} at 127.28.17.7:35785
I20240501 14:00:39.369585 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-98e027e2a36541eda1a67c0a3ab48d65 on tablet 98e027e2a36541eda1a67c0a3ab48d65 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:39.374428 2143 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:39.375414 4901 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:39.378544 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 8afb34c628104a51a0d9c9d4dd649123 (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.385107 1999 catalog_manager.cc:5484] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:39.396068 5392 ts_tablet_manager.cc:1888] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:39.397843 5392 log.cc:1196] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/8afb34c628104a51a0d9c9d4dd649123
I20240501 14:00:39.399277 5392 ts_tablet_manager.cc:1909] T 8afb34c628104a51a0d9c9d4dd649123 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:39.401692 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet a934e21a102e42658beca7732818202d (table rebalancer_dynamic_table_new_name [id=95d8a18f01744ad8b66ed5b8f0ec430e]) successfully deleted
I20240501 14:00:39.401950 5389 tablet_replica.cc:330] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:39.402209 5389 raft_consensus.cc:2229] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.402547 5389 raft_consensus.cc:2258] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.402946 2278 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:39.403399 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 8afb34c628104a51a0d9c9d4dd649123 (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.404083 5392 tablet_replica.cc:330] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:39.404181 5944 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.404599 5392 raft_consensus.cc:2229] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.404706 5944 tablet_bootstrap.cc:492] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:39.404999 5944 ts_tablet_manager.cc:1367] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.141s user 0.041s sys 0.009s
I20240501 14:00:39.405613 5392 raft_consensus.cc:2258] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.405822 5944 raft_consensus.cc:348] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:39.406100 5944 raft_consensus.cc:729] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:39.407156 5389 ts_tablet_manager.cc:1875] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.406314 5944 consensus_queue.cc:260] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:39.408026 5392 ts_tablet_manager.cc:1875] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.412151 5944 ts_tablet_manager.cc:1398] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:00:39.412987 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-87a79bfca7cd4ffcbb0e4448320d0576 received from {username='slave'} at 127.28.17.5:50935
I20240501 14:00:39.413112 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-87a79bfca7cd4ffcbb0e4448320d0576 on tablet 87a79bfca7cd4ffcbb0e4448320d0576 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:39.420982 5851 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.421435 5851 tablet_bootstrap.cc:492] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:00:39.421708 5851 ts_tablet_manager.cc:1367] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.146s user 0.042s sys 0.008s
I20240501 14:00:39.422592 5851 raft_consensus.cc:348] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:39.422852 5851 raft_consensus.cc:729] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:00:39.423061 5851 consensus_queue.cc:260] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:39.423620 5851 ts_tablet_manager.cc:1398] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:39.423910 5389 ts_tablet_manager.cc:1888] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:39.424103 5389 log.cc:1196] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/e58200ed1d53430e97ed77e154dee4a7
I20240501 14:00:39.424559 5389 ts_tablet_manager.cc:1909] T e58200ed1d53430e97ed77e154dee4a7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:39.425601 5389 tablet_replica.cc:330] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:39.425673 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet e58200ed1d53430e97ed77e154dee4a7 (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.425971 5389 raft_consensus.cc:2229] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.426321 5389 raft_consensus.cc:2258] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.427084 5392 ts_tablet_manager.cc:1888] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:39.427228 5392 log.cc:1196] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/a131d188420344e0a76c7b84596194ea
I20240501 14:00:39.427800 5389 ts_tablet_manager.cc:1875] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.429111 5392 ts_tablet_manager.cc:1909] T a131d188420344e0a76c7b84596194ea P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:39.430105 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet a131d188420344e0a76c7b84596194ea (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.432893 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-85842504df3e4571b1a9750809711db2 received from {username='slave'} at 127.28.17.8:39671
I20240501 14:00:39.433019 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-85842504df3e4571b1a9750809711db2 on tablet 85842504df3e4571b1a9750809711db2 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:39.434866 5389 ts_tablet_manager.cc:1888] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:39.435053 5389 log.cc:1196] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/a131d188420344e0a76c7b84596194ea
I20240501 14:00:39.435570 5389 ts_tablet_manager.cc:1909] T a131d188420344e0a76c7b84596194ea P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:39.436609 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet a131d188420344e0a76c7b84596194ea (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.436978 5389 tablet_replica.cc:330] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:39.437261 5389 raft_consensus.cc:2229] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.437546 5389 raft_consensus.cc:2258] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.438658 5389 ts_tablet_manager.cc:1875] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:39.445359 5389 ts_tablet_manager.cc:1888] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:00:39.445628 5389 log.cc:1196] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/8afb34c628104a51a0d9c9d4dd649123
I20240501 14:00:39.446101 5389 ts_tablet_manager.cc:1909] T 8afb34c628104a51a0d9c9d4dd649123 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:39.447206 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 8afb34c628104a51a0d9c9d4dd649123 (table rebalance_test_table_18 [id=217e764ff5274de0acedaa6bd366d8a6]) successfully deleted
I20240501 14:00:39.450537 5964 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.460880 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:39.475046 5938 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.476621 5938 tablet_bootstrap.cc:492] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:39.477598 5938 ts_tablet_manager.cc:1367] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.233s user 0.082s sys 0.011s
I20240501 14:00:39.478638 4900 raft_consensus.cc:1203] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.479655 5938 raft_consensus.cc:348] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:39.480427 5938 raft_consensus.cc:729] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:39.480831 5938 consensus_queue.cc:260] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:39.482523 5938 ts_tablet_manager.cc:1398] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:39.483754 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-5b1f151470f24501857f38461d5b70c8 received from {username='slave'} at 127.28.17.6:41439
I20240501 14:00:39.483887 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-5b1f151470f24501857f38461d5b70c8 on tablet 5b1f151470f24501857f38461d5b70c8 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:39.493191 5964 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.543016 2406 raft_consensus.cc:1261] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:39.544692 5994 consensus_queue.cc:1035] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.554416 2536 raft_consensus.cc:1261] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:39.555408 5994 consensus_queue.cc:1035] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.568626 1999 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet c32a8d5414fb4364a63004c6c6eafe1a (table rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56]). Expected version 1 got 0
I20240501 14:00:39.570909 6023 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:39.572765 6023 tablet.cc:1620] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:39.573379 1999 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet c32a8d5414fb4364a63004c6c6eafe1a (table rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56]). Expected version 1 got 0
I20240501 14:00:39.577466 6022 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:39.578922 6022 tablet.cc:1620] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:39.579505 2003 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet c32a8d5414fb4364a63004c6c6eafe1a (table rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56]). Expected version 1 got 0
I20240501 14:00:39.589190 4901 raft_consensus.cc:1203] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.590126 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56] alter complete (version 1)
I20240501 14:00:39.602666 6024 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:39.604071 6024 tablet.cc:1620] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:39.605680 2003 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet c32a8d5414fb4364a63004c6c6eafe1a (table rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56]). Expected version 1 got 0
I20240501 14:00:39.612957 5867 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.613350 5867 tablet_bootstrap.cc:492] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:39.613610 5867 ts_tablet_manager.cc:1367] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.829s user 0.061s sys 0.000s
I20240501 14:00:39.614241 5867 raft_consensus.cc:348] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:39.614454 5867 raft_consensus.cc:729] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:39.616531 5867 consensus_queue.cc:260] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:39.617977 5867 ts_tablet_manager.cc:1398] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 14:00:39.619387 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-fac8d3842d994279b355f9a18a82f901 received from {username='slave'} at 127.28.17.7:46539
I20240501 14:00:39.619516 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-fac8d3842d994279b355f9a18a82f901 on tablet fac8d3842d994279b355f9a18a82f901 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:39.637040 4800 consensus_queue.cc:1035] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.665246 5713 consensus_queue.cc:1035] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.688830 6041 ts_tablet_manager.cc:902] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:39.696517 5617 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.698930 5617 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:39.700506 6044 raft_consensus.cc:1050] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:00:39.701977 6044 consensus_queue.cc:237] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:39.702915 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:39.707037 5163 raft_consensus.cc:1203] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.707226 2006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:39.714296 5296 raft_consensus.cc:1203] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.717689 5163 raft_consensus.cc:1203] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.724910 6022 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:39.725420 6022 tablet.cc:1620] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:39.727502 6024 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:39.728013 6024 tablet.cc:1620] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:39.733464 6023 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:39.733974 6023 tablet.cc:1620] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:39.743481 5295 raft_consensus.cc:1261] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:39.744208 5958 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:39.745117 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 6904a974640a4e84a80b1fb0749365b7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.745584 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6904a974640a4e84a80b1fb0749365b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.747290 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 0fac2baeae824f68ae2fe1c62e36d970 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.747557 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 6904a974640a4e84a80b1fb0749365b7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.747773 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fac2baeae824f68ae2fe1c62e36d970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.747983 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6904a974640a4e84a80b1fb0749365b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.748786 4879 tablet_service.cc:1467] Processing CreateTablet for tablet d841d51e617d44c8adf08658c69d957b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.752682 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 0fac2baeae824f68ae2fe1c62e36d970 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.753188 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fac2baeae824f68ae2fe1c62e36d970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.755049 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d841d51e617d44c8adf08658c69d957b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.758257 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 6904a974640a4e84a80b1fb0749365b7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.758306 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 0fac2baeae824f68ae2fe1c62e36d970 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.758738 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6904a974640a4e84a80b1fb0749365b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.758769 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fac2baeae824f68ae2fe1c62e36d970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.760381 5294 raft_consensus.cc:1203] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.762852 5275 tablet_service.cc:1467] Processing CreateTablet for tablet d841d51e617d44c8adf08658c69d957b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.763279 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d841d51e617d44c8adf08658c69d957b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.763036 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=2e6703fd607845a797e25c6fb3ef9c56] alter complete (version 2)
I20240501 14:00:39.766956 5012 tablet_service.cc:1467] Processing CreateTablet for tablet d841d51e617d44c8adf08658c69d957b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:39.767405 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d841d51e617d44c8adf08658c69d957b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.789547 6056 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:39.791402 6056 tablet_bootstrap.cc:654] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.794073 2276 raft_consensus.cc:1261] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:39.794796 2536 raft_consensus.cc:1261] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:39.795694 6055 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:39.795663 5958 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:39.796303 5958 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:39.797443 6055 tablet_bootstrap.cc:654] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.800654 6054 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:39.802641 6054 tablet_bootstrap.cc:654] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.804471 6059 tablet_bootstrap.cc:492] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:39.806293 6059 tablet_bootstrap.cc:654] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.807955 6056 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:39.808908 6056 ts_tablet_manager.cc:1367] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 14:00:39.811383 6041 tablet_copy_client.cc:308] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:39.812352 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 052964af500c427e836f0f3121aa91f3 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35609)
I20240501 14:00:39.812538 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 052964af500c427e836f0f3121aa91f3 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35609: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-052964af500c427e836f0f3121aa91f3
I20240501 14:00:39.815711 2558 tablet_copy_source_session.cc:205] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:39.817790 6041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052964af500c427e836f0f3121aa91f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:39.826030 6056 raft_consensus.cc:348] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:39.826423 6056 raft_consensus.cc:374] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.826607 6056 raft_consensus.cc:729] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.826973 6056 consensus_queue.cc:260] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:39.828581 6056 ts_tablet_manager.cc:1398] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:39.828927 6056 tablet_bootstrap.cc:492] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:39.832741 6054 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:39.833387 6054 ts_tablet_manager.cc:1367] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.004s
I20240501 14:00:39.834334 6054 raft_consensus.cc:348] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:39.834656 6054 raft_consensus.cc:374] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.834791 6054 raft_consensus.cc:729] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.835065 6054 consensus_queue.cc:260] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:39.835723 6054 ts_tablet_manager.cc:1398] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:39.837224 6059 tablet_bootstrap.cc:492] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:39.837641 6059 ts_tablet_manager.cc:1367] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240501 14:00:39.838492 6059 raft_consensus.cc:348] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.839646 6059 raft_consensus.cc:374] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.839787 6059 raft_consensus.cc:729] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.840148 6059 consensus_queue.cc:260] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.841614 6055 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:39.841827 6055 ts_tablet_manager.cc:1367] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240501 14:00:39.842777 6055 raft_consensus.cc:348] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:39.843044 6055 raft_consensus.cc:374] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.843179 6055 raft_consensus.cc:729] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.843510 6055 consensus_queue.cc:260] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:39.844916 6055 ts_tablet_manager.cc:1398] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:39.845584 6059 ts_tablet_manager.cc:1398] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:39.845961 6059 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:39.848929 6056 tablet_bootstrap.cc:654] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.852488 6041 tablet_copy_client.cc:783] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:39.853353 6041 tablet_copy_client.cc:647] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:39.824798 6060 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:39.856652 5958 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:39.857149 6041 tablet_copy_client.cc:523] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:39.859951 6059 tablet_bootstrap.cc:654] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.867233 2000 catalog_manager.cc:5484] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:39.868561 5295 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:39.877643 2536 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:39.879317 2276 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:39.882092 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:39.882961 6057 tablet_bootstrap.cc:492] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:39.883615 6059 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:39.883880 6059 ts_tablet_manager.cc:1367] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.003s
I20240501 14:00:39.884781 6059 raft_consensus.cc:348] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.880364 6041 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:39.885047 6060 tablet_bootstrap.cc:654] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.886714 6057 tablet_bootstrap.cc:654] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.890019 6056 tablet_bootstrap.cc:492] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:39.890264 6056 ts_tablet_manager.cc:1367] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.000s
I20240501 14:00:39.891068 6056 raft_consensus.cc:348] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.891379 6056 raft_consensus.cc:374] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.891490 6056 raft_consensus.cc:729] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.891782 6056 consensus_queue.cc:260] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.892709 2147 consensus_queue.cc:237] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:39.893442 6056 ts_tablet_manager.cc:1398] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:39.894083 6059 raft_consensus.cc:374] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.894251 6059 raft_consensus.cc:729] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.894656 6059 consensus_queue.cc:260] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.896241 6059 ts_tablet_manager.cc:1398] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:00:39.897609 5295 raft_consensus.cc:1261] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:39.898313 6062 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:39.901769 5295 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:39.903406 6022 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:39.906006 6044 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:39.908262 6023 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:39.908766 6041 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:39.909862 6041 tablet_bootstrap.cc:492] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:00:39.910254 6041 ts_tablet_manager.cc:1367] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.001s
I20240501 14:00:39.911295 6041 raft_consensus.cc:348] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:00:39.911553 6041 raft_consensus.cc:729] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:00:39.911780 6041 consensus_queue.cc:260] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:00:39.912626 6041 ts_tablet_manager.cc:1398] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:39.913455 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-052964af500c427e836f0f3121aa91f3 received from {username='slave'} at 127.28.17.2:35609
I20240501 14:00:39.913589 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-052964af500c427e836f0f3121aa91f3 on tablet 052964af500c427e836f0f3121aa91f3 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:39.914170 2000 catalog_manager.cc:5484] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:39.915467 6024 tablet.cc:1722] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:39.916785 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6c38a8bcb4a4e07aa424fd074d5e17e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:39.918175 2276 raft_consensus.cc:1261] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:39.920644 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c38a8bcb4a4e07aa424fd074d5e17e with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:39.920574 6062 consensus_queue.cc:1035] T b6c38a8bcb4a4e07aa424fd074d5e17e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 14:00:39.927161 6080 tablet_replica.cc:330] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:39.927421 6080 raft_consensus.cc:2229] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:39.927762 6080 raft_consensus.cc:2258] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:39.928383 2276 raft_consensus.cc:2932] T b6c38a8bcb4a4e07aa424fd074d5e17e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:39.930255 6080 ts_tablet_manager.cc:1875] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:39.935730 6057 tablet_bootstrap.cc:492] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:39.936264 6057 ts_tablet_manager.cc:1367] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.000s
I20240501 14:00:39.937332 6057 raft_consensus.cc:348] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.937631 6057 raft_consensus.cc:374] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.937777 6057 raft_consensus.cc:729] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.938097 6057 consensus_queue.cc:260] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.938745 5034 raft_consensus.cc:1203] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:39.941556 6057 ts_tablet_manager.cc:1398] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:39.942673 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=2e6703fd607845a797e25c6fb3ef9c56] alter complete (version 3)
I20240501 14:00:39.944831 6080 ts_tablet_manager.cc:1888] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:39.945060 6080 log.cc:1196] T b6c38a8bcb4a4e07aa424fd074d5e17e P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/b6c38a8bcb4a4e07aa424fd074d5e17e
I20240501 14:00:39.946787 6060 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:39.946818 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet b6c38a8bcb4a4e07aa424fd074d5e17e (table rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]) successfully deleted
I20240501 14:00:39.947512 6060 ts_tablet_manager.cc:1367] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.123s user 0.008s sys 0.000s
I20240501 14:00:39.948325 6060 raft_consensus.cc:348] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.949491 6060 raft_consensus.cc:374] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.949601 6060 raft_consensus.cc:729] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.949885 6060 consensus_queue.cc:260] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.950657 6060 ts_tablet_manager.cc:1398] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:39.956512 6057 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:39.959226 6057 tablet_bootstrap.cc:654] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:39.967831 6057 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:39.968122 6057 ts_tablet_manager.cc:1367] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:00:39.969499 6057 raft_consensus.cc:348] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.969698 6057 raft_consensus.cc:374] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:39.969799 6057 raft_consensus.cc:729] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:39.970031 6057 consensus_queue.cc:260] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.970708 6057 ts_tablet_manager.cc:1398] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:39.986675 5994 raft_consensus.cc:482] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:39.986918 5994 raft_consensus.cc:504] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:39.987866 5994 leader_election.cc:288] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:40.005053 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d841d51e617d44c8adf08658c69d957b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:40.005336 4901 raft_consensus.cc:2454] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:40.006757 5229 leader_election.cc:302] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:40.007184 5994 raft_consensus.cc:2781] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:40.007285 5994 raft_consensus.cc:482] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:40.007385 5994 raft_consensus.cc:3037] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.010195 5994 raft_consensus.cc:504] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.011509 5994 leader_election.cc:288] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:40.011687 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d841d51e617d44c8adf08658c69d957b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:40.011904 4901 raft_consensus.cc:3037] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.014394 4901 raft_consensus.cc:2454] T d841d51e617d44c8adf08658c69d957b P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:40.015067 5229 leader_election.cc:302] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:40.015427 5994 raft_consensus.cc:2781] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:40.015597 5994 raft_consensus.cc:686] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:40.015867 5994 consensus_queue.cc:237] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.016184 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d841d51e617d44c8adf08658c69d957b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:40.016268 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d841d51e617d44c8adf08658c69d957b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:40.016475 5034 raft_consensus.cc:2454] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:40.020713 1999 catalog_manager.cc:5484] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:40.030406 5013 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:40.033635 6062 raft_consensus.cc:482] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:40.033813 6062 raft_consensus.cc:504] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:40.034737 6062 leader_election.cc:288] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:40.035213 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fac2baeae824f68ae2fe1c62e36d970" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:40.035521 5034 raft_consensus.cc:2454] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:40.036082 2082 leader_election.cc:302] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:40.036489 6062 raft_consensus.cc:2781] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:40.036329 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fac2baeae824f68ae2fe1c62e36d970" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:40.036607 6062 raft_consensus.cc:482] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:40.036688 6062 raft_consensus.cc:3037] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.036685 5163 raft_consensus.cc:2454] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:40.039549 6062 raft_consensus.cc:504] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:40.041033 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fac2baeae824f68ae2fe1c62e36d970" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:40.041260 5034 raft_consensus.cc:3037] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.041603 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fac2baeae824f68ae2fe1c62e36d970" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:40.041798 5163 raft_consensus.cc:3037] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.044315 5034 raft_consensus.cc:2454] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:40.044981 2082 leader_election.cc:302] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:40.045084 5163 raft_consensus.cc:2454] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:40.045287 6044 raft_consensus.cc:2781] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:40.045410 6062 leader_election.cc:288] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:40.045496 6044 raft_consensus.cc:686] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:40.045776 6044 consensus_queue.cc:237] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:40.048745 1999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:40.049005 1999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:40.053063 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet c32a8d5414fb4364a63004c6c6eafe1a
I20240501 14:00:40.053186 5275 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:00:40.053906 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet c32a8d5414fb4364a63004c6c6eafe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:40 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:40.057924 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet c32a8d5414fb4364a63004c6c6eafe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:40 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:40.058284 6080 tablet_replica.cc:330] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:40.058543 6080 raft_consensus.cc:2229] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.058777 6080 raft_consensus.cc:2258] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.059819 6080 ts_tablet_manager.cc:1875] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:40.060781 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet c32a8d5414fb4364a63004c6c6eafe1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:40 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:40.066874 6080 ts_tablet_manager.cc:1888] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:40.067124 6080 log.cc:1196] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/c32a8d5414fb4364a63004c6c6eafe1a
I20240501 14:00:40.067618 6080 ts_tablet_manager.cc:1909] T c32a8d5414fb4364a63004c6c6eafe1a P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:40.068890 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet c32a8d5414fb4364a63004c6c6eafe1a (table rebalancer_dynamic_table_new_name [id=2e6703fd607845a797e25c6fb3ef9c56]) successfully deleted
I20240501 14:00:40.082145 5994 raft_consensus.cc:482] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:40.082413 5994 raft_consensus.cc:504] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.082927 6095 tablet_replica.cc:330] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:40.083997 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6904a974640a4e84a80b1fb0749365b7" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:40.084272 4901 raft_consensus.cc:2454] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:40.084179 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6904a974640a4e84a80b1fb0749365b7" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:40.084442 2406 raft_consensus.cc:2454] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:40.083305 5994 leader_election.cc:288] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:40.084895 5229 leader_election.cc:302] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:40.085245 6095 raft_consensus.cc:2229] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:40.085781 5994 raft_consensus.cc:2781] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:40.085842 6095 raft_consensus.cc:2258] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.085919 5994 raft_consensus.cc:482] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:40.086001 5994 raft_consensus.cc:3037] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.088766 2276 raft_consensus.cc:1203] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:40.089161 5994 raft_consensus.cc:504] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.091048 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6904a974640a4e84a80b1fb0749365b7" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:40.091065 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6904a974640a4e84a80b1fb0749365b7" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:40.091302 2406 raft_consensus.cc:3037] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.091320 4901 raft_consensus.cc:3037] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.092837 6097 tablet_replica.cc:330] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:40.093058 6097 raft_consensus.cc:2229] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.093291 6097 raft_consensus.cc:2258] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.093995 4901 raft_consensus.cc:2454] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:40.094763 5229 leader_election.cc:302] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:40.094885 2406 raft_consensus.cc:2454] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:40.095151 5997 raft_consensus.cc:2781] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:40.095301 5997 raft_consensus.cc:686] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:40.095631 5997 consensus_queue.cc:237] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.098098 6095 ts_tablet_manager.cc:1875] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:40.099457 5994 leader_election.cc:288] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:40.100557 1999 catalog_manager.cc:5484] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:40.101778 6097 ts_tablet_manager.cc:1875] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:40.113394 5976 raft_consensus.cc:1050] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:40.114612 5976 consensus_queue.cc:237] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:40.117307 4901 raft_consensus.cc:1261] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.117945 5727 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.120992 2406 raft_consensus.cc:1261] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.120975 2147 raft_consensus.cc:1261] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.125717 6027 raft_consensus.cc:1050] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:40.126824 6027 consensus_queue.cc:237] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:40.128790 5967 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.129258 5976 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.147543 1999 catalog_manager.cc:5484] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:40.157820 5976 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.162801 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:40.164146 4901 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.165228 2407 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.166045 2127 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:00:40.170811 4900 raft_consensus.cc:1261] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.171747 2001 catalog_manager.cc:5484] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.172251 5507 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.175532 6095 ts_tablet_manager.cc:1888] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:40.177755 2538 raft_consensus.cc:1261] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.178107 6095 log.cc:1196] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/c32a8d5414fb4364a63004c6c6eafe1a
I20240501 14:00:40.178864 6095 ts_tablet_manager.cc:1909] T c32a8d5414fb4364a63004c6c6eafe1a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:40.180178 5507 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.181074 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 68027870fbb9438fb857c1ad55311a41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd]), partition=
I20240501 14:00:40.181130 6026 raft_consensus.cc:1050] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:00:40.181473 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68027870fbb9438fb857c1ad55311a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.181962 6026 consensus_queue.cc:237] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:40.183022 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet c32a8d5414fb4364a63004c6c6eafe1a (table rebalancer_dynamic_table_new_name [id=2e6703fd607845a797e25c6fb3ef9c56]) successfully deleted
I20240501 14:00:40.185375 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 68027870fbb9438fb857c1ad55311a41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd]), partition=
I20240501 14:00:40.187366 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 68027870fbb9438fb857c1ad55311a41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd]), partition=
I20240501 14:00:40.187690 2406 raft_consensus.cc:1261] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.187738 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68027870fbb9438fb857c1ad55311a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.189536 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68027870fbb9438fb857c1ad55311a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.194191 6059 tablet_bootstrap.cc:492] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:40.195991 6059 tablet_bootstrap.cc:654] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.198148 6027 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.203379 5727 raft_consensus.cc:1050] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:40.204246 5727 consensus_queue.cc:237] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:40.213068 5163 raft_consensus.cc:1261] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.213618 6117 tablet_bootstrap.cc:492] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:40.213897 5976 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.215448 6117 tablet_bootstrap.cc:654] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.218161 6057 tablet_bootstrap.cc:492] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:40.220342 6057 tablet_bootstrap.cc:654] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.231514 6059 tablet_bootstrap.cc:492] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:40.231776 6059 ts_tablet_manager.cc:1367] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.004s
I20240501 14:00:40.232658 2276 raft_consensus.cc:1261] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.232656 6059 raft_consensus.cc:348] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.233064 6059 raft_consensus.cc:374] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.233207 6059 raft_consensus.cc:729] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.233525 6059 consensus_queue.cc:260] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.233676 5727 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.234247 5142 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:00:40.239099 6059 ts_tablet_manager.cc:1398] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:00:40.242247 6097 ts_tablet_manager.cc:1888] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:40.243283 6097 log.cc:1196] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/c32a8d5414fb4364a63004c6c6eafe1a
I20240501 14:00:40.245323 6117 tablet_bootstrap.cc:492] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:40.245617 6117 ts_tablet_manager.cc:1367] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.007s
I20240501 14:00:40.246435 6117 raft_consensus.cc:348] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.247054 6117 raft_consensus.cc:374] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.247196 6117 raft_consensus.cc:729] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.247467 6117 consensus_queue.cc:260] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.249503 6117 ts_tablet_manager.cc:1398] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:40.262131 2146 raft_consensus.cc:1261] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.263835 5976 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.265259 2143 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.269198 2535 consensus_queue.cc:237] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:40.277648 2143 raft_consensus.cc:1261] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.277712 4900 raft_consensus.cc:1261] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.278198 5727 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.278707 5727 consensus_queue.cc:1035] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.282606 2143 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.281548 5967 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.289727 6115 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.291002 4901 raft_consensus.cc:2932] T 87a79bfca7cd4ffcbb0e4448320d0576 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.295301 2144 raft_consensus.cc:1261] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.295964 5295 raft_consensus.cc:1261] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.295974 6115 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.296715 5507 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.296831 4900 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.299285 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87a79bfca7cd4ffcbb0e4448320d0576 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.303097 1997 catalog_manager.cc:5484] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:40.304368 6057 tablet_bootstrap.cc:492] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:40.304535 6057 ts_tablet_manager.cc:1367] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.004s
I20240501 14:00:40.306231 2516 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:40.306859 6057 raft_consensus.cc:348] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.305733 2538 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.309700 6057 raft_consensus.cc:374] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.311033 6057 raft_consensus.cc:729] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.311929 6057 consensus_queue.cc:260] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.301721 4799 raft_consensus.cc:1050] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:00:40.318239 4799 consensus_queue.cc:237] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:40.308892 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a79bfca7cd4ffcbb0e4448320d0576 with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:40.321960 2001 catalog_manager.cc:5484] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:40.324084 6115 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.324357 2406 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.326756 2147 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.331789 2536 raft_consensus.cc:1261] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.334888 5507 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.335789 2278 consensus_queue.cc:237] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:40.337858 5727 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.338543 2001 catalog_manager.cc:5484] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.345047 5163 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.348579 4800 raft_consensus.cc:1050] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:40.349613 4800 consensus_queue.cc:237] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:40.350759 2387 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:00:40.353756 1999 catalog_manager.cc:5484] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.356783 2278 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.357674 5295 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.362596 6097 ts_tablet_manager.cc:1909] T c32a8d5414fb4364a63004c6c6eafe1a P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:40.356748 4901 raft_consensus.cc:1261] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.364511 6130 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.368743 2537 consensus_queue.cc:237] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:40.372658 2258 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:00:40.373909 5163 raft_consensus.cc:1261] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.375312 5967 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.377990 5976 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.379612 2278 raft_consensus.cc:1261] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.380312 5507 raft_consensus.cc:482] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:40.380523 5507 raft_consensus.cc:504] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.381850 5507 leader_election.cc:288] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:40.381857 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68027870fbb9438fb857c1ad55311a41" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:40.382097 5295 raft_consensus.cc:2454] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:40.382616 2213 leader_election.cc:302] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:40.382907 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98e027e2a36541eda1a67c0a3ab48d65 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.383024 5507 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.383913 5976 consensus_queue.cc:1035] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.386545 6027 raft_consensus.cc:2781] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:40.387360 5163 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.392052 2278 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.392072 2537 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.392390 6027 raft_consensus.cc:482] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:40.392508 6027 raft_consensus.cc:3037] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.392792 4799 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.392915 5295 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.393093 6132 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.393534 6097 tablet_replica.cc:330] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:40.393527 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet c32a8d5414fb4364a63004c6c6eafe1a (table rebalancer_dynamic_table_new_name [id=2e6703fd607845a797e25c6fb3ef9c56]) successfully deleted
I20240501 14:00:40.393858 6097 raft_consensus.cc:2229] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.393790 6132 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:00:40.394412 2211 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.4:36117, user_credentials={real_user=slave}} blocked reactor thread for 61260.4us
I20240501 14:00:40.372853 6057 ts_tablet_manager.cc:1398] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20240501 14:00:40.400230 6027 raft_consensus.cc:504] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.402822 6097 raft_consensus.cc:2258] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.406944 2000 catalog_manager.cc:5484] T 98e027e2a36541eda1a67c0a3ab48d65 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.389079 4900 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.388252 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68027870fbb9438fb857c1ad55311a41" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:40.411011 4901 raft_consensus.cc:2454] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:40.413736 6097 ts_tablet_manager.cc:1875] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.413256 2276 raft_consensus.cc:2932] T 98e027e2a36541eda1a67c0a3ab48d65 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.414422 6027 leader_election.cc:288] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:40.414863 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68027870fbb9438fb857c1ad55311a41" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:40.415069 5296 raft_consensus.cc:3037] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.414321 2534 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.417726 5296 raft_consensus.cc:2454] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:40.418452 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b3738397b81b46868e09071be6f82334 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.418991 5034 raft_consensus.cc:3037] T d841d51e617d44c8adf08658c69d957b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.418978 2213 leader_election.cc:302] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:40.419517 6131 raft_consensus.cc:2781] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:40.419673 6131 raft_consensus.cc:686] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:40.419960 6131 consensus_queue.cc:237] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.421111 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68027870fbb9438fb857c1ad55311a41" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:40.421811 4900 raft_consensus.cc:3037] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.422678 5994 consensus_queue.cc:1035] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.429474 2001 catalog_manager.cc:5484] T b3738397b81b46868e09071be6f82334 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.433248 2275 consensus_queue.cc:237] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:40.440145 2536 raft_consensus.cc:1261] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.441017 5296 raft_consensus.cc:1261] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.441135 2143 raft_consensus.cc:1261] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.441813 6026 consensus_queue.cc:1035] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.443572 2536 raft_consensus.cc:2932] T b3738397b81b46868e09071be6f82334 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:40.447986 5713 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.442512 6026 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20240501 14:00:40.451413 5507 consensus_queue.cc:1035] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.452853 4900 raft_consensus.cc:2454] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:40.453778 2275 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.454532 5296 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.458038 2143 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.453007 6115 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.468616 4879 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:00:40.471246 2143 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:00:40.471843 6137 log.cc:924] Time spent T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20240501 14:00:40.471801 6132 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.462131 2006 catalog_manager.cc:5484] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.480307 5163 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.481396 6132 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.464855 1999 catalog_manager.cc:5484] T 85842504df3e4571b1a9750809711db2 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.466650 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85842504df3e4571b1a9750809711db2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.483595 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 98e027e2a36541eda1a67c0a3ab48d65 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:40.483428 2003 catalog_manager.cc:5484] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:40.485390 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet b3738397b81b46868e09071be6f82334 with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:40.489526 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 85842504df3e4571b1a9750809711db2 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:40.489953 6080 tablet_replica.cc:330] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:40.490187 6080 raft_consensus.cc:2229] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.490473 6080 raft_consensus.cc:2258] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.492092 5296 raft_consensus.cc:2932] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.495203 6080 ts_tablet_manager.cc:1875] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.497320 6097 ts_tablet_manager.cc:1888] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:40.497483 6097 log.cc:1196] T 87a79bfca7cd4ffcbb0e4448320d0576 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/87a79bfca7cd4ffcbb0e4448320d0576
I20240501 14:00:40.502357 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 87a79bfca7cd4ffcbb0e4448320d0576 (table rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]) successfully deleted
I20240501 14:00:40.503417 6097 tablet_replica.cc:330] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:40.504021 6097 raft_consensus.cc:2229] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.504585 6044 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.474942 2536 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.506748 6097 raft_consensus.cc:2258] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.511713 6097 ts_tablet_manager.cc:1875] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.517619 2534 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.518456 6141 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.519490 5727 raft_consensus.cc:1050] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:00:40.520350 5727 consensus_queue.cc:237] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:00:40.528295 6080 ts_tablet_manager.cc:1888] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:40.529302 6132 raft_consensus.cc:1050] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:40.530169 6132 consensus_queue.cc:237] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:40.530908 6080 log.cc:1196] T 85842504df3e4571b1a9750809711db2 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/85842504df3e4571b1a9750809711db2
I20240501 14:00:40.532459 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 85842504df3e4571b1a9750809711db2 (table rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]) successfully deleted
I20240501 14:00:40.533028 5796 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.534713 6147 tablet_replica.cc:330] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:40.537663 5164 raft_consensus.cc:1261] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:40.538379 5976 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:40.539686 5296 raft_consensus.cc:1261] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:40.540449 2278 raft_consensus.cc:1261] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:40.540983 6147 raft_consensus.cc:2229] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.541055 5976 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:40.541671 5976 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:40.542182 6147 pending_rounds.cc:62] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Trying to abort 1 pending ops.
I20240501 14:00:40.543076 6147 pending_rounds.cc:68] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022887076673871872 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "98e027e2a36541eda1a67c0a3ab48d65" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } } }
I20240501 14:00:40.545794 4799 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.548444 2535 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.553193 5163 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.555343 1999 catalog_manager.cc:5484] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.556038 6147 raft_consensus.cc:2894] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:00:40.556459 6147 raft_consensus.cc:2866] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:00:40.556828 6147 raft_consensus.cc:2258] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.554231 2147 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.563473 5976 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:40.562170 6147 ts_tablet_manager.cc:1875] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.567711 6044 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.570029 5162 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:40.570703 2143 raft_consensus.cc:1261] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.571385 5032 raft_consensus.cc:1261] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.572043 4799 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:00:40.572605 6132 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.576241 5997 consensus_queue.cc:1035] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.576020 2000 catalog_manager.cc:5484] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:40.578927 6097 ts_tablet_manager.cc:1888] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:40.579080 6097 log.cc:1196] T b3738397b81b46868e09071be6f82334 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/b3738397b81b46868e09071be6f82334
I20240501 14:00:40.580011 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet b3738397b81b46868e09071be6f82334 (table rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]) successfully deleted
I20240501 14:00:40.591223 2275 raft_consensus.cc:1261] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:40.567194 2278 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:40.593808 6141 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:40.595237 2534 consensus_queue.cc:237] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:00:40.599675 2276 raft_consensus.cc:1261] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:40.590276 5296 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:40.600788 5976 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:40.603924 5967 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:40.604511 2278 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:40.605114 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:40.607656 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 052964af500c427e836f0f3121aa91f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.609838 6147 ts_tablet_manager.cc:1888] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:40.609978 6147 log.cc:1196] T 98e027e2a36541eda1a67c0a3ab48d65 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/98e027e2a36541eda1a67c0a3ab48d65
I20240501 14:00:40.611701 2001 catalog_manager.cc:5484] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:40.612063 5796 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.617933 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 052964af500c427e836f0f3121aa91f3 with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:40.631873 4899 raft_consensus.cc:1261] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:40.632964 6027 consensus_queue.cc:1035] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.633394 6168 tablet_replica.cc:330] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:40.633635 6168 raft_consensus.cc:2229] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.633898 6168 raft_consensus.cc:2258] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.634141 5293 raft_consensus.cc:1261] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:40.634903 6115 consensus_queue.cc:1035] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:40.636482 6168 ts_tablet_manager.cc:1875] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.643129 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 98e027e2a36541eda1a67c0a3ab48d65 (table rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]) successfully deleted
I20240501 14:00:40.648630 5295 raft_consensus.cc:1261] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:40.651111 5976 consensus_queue.cc:1035] T 052964af500c427e836f0f3121aa91f3 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:40.652863 5295 raft_consensus.cc:2932] T 052964af500c427e836f0f3121aa91f3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:40.653096 4800 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.658668 2147 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.659636 6168 ts_tablet_manager.cc:1888] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:40.659832 6168 log.cc:1196] T 052964af500c427e836f0f3121aa91f3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/052964af500c427e836f0f3121aa91f3
I20240501 14:00:40.661739 2278 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.662698 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 052964af500c427e836f0f3121aa91f3 (table rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]) successfully deleted
I20240501 14:00:40.662796 5032 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.664769 2001 catalog_manager.cc:5484] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.683271 2407 consensus_queue.cc:237] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:40.685720 5295 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.686484 6132 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.689373 6132 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.693748 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1e048df08cf4f7eaacb78d6592d79b1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.695309 5295 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.697994 2001 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 68027870fbb9438fb857c1ad55311a41 (table rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd]). Expected version 1 got 0
I20240501 14:00:40.702733 2275 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.703370 5713 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.705775 2275 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.710461 1997 catalog_manager.cc:5484] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:40.717803 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e048df08cf4f7eaacb78d6592d79b1 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:40.718240 6080 tablet_replica.cc:330] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:40.718482 6080 raft_consensus.cc:2229] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.718772 6080 raft_consensus.cc:2258] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.720064 6080 ts_tablet_manager.cc:1875] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.723380 2406 consensus_queue.cc:237] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:40.725211 5164 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.725564 2534 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.726365 4800 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.727963 6142 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.729271 2407 consensus_queue.cc:237] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:40.729635 6080 ts_tablet_manager.cc:1888] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:40.729771 6080 log.cc:1196] T f1e048df08cf4f7eaacb78d6592d79b1 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/f1e048df08cf4f7eaacb78d6592d79b1
I20240501 14:00:40.730815 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet f1e048df08cf4f7eaacb78d6592d79b1 (table rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]) successfully deleted
I20240501 14:00:40.731369 4800 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.736312 5033 raft_consensus.cc:1261] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.736961 6142 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.737002 5164 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.737041 2534 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.739418 6132 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.740125 5033 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.736351 2001 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:40.743973 2006 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 68027870fbb9438fb857c1ad55311a41 (table rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd]). Expected version 1 got 0
I20240501 14:00:40.746837 6175 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:40.748168 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b1f151470f24501857f38461d5b70c8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.750126 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fac8d3842d994279b355f9a18a82f901 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:40.748170 2003 catalog_manager.cc:5484] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:40.751103 2147 raft_consensus.cc:1261] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:40.751735 5713 consensus_queue.cc:1035] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:40.753298 2147 raft_consensus.cc:2932] T 5b1f151470f24501857f38461d5b70c8 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:40.754442 1997 catalog_manager.cc:5484] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:40.755940 4879 tablet_service.cc:1467] Processing CreateTablet for tablet c565cfcf616445ff949dd75f8f513e88 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.756526 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c565cfcf616445ff949dd75f8f513e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.756508 5275 tablet_service.cc:1467] Processing CreateTablet for tablet c565cfcf616445ff949dd75f8f513e88 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.756820 5274 tablet_service.cc:1467] Processing CreateTablet for tablet c88cc21c0e0743999ee493f5ee856aef (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.756927 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c565cfcf616445ff949dd75f8f513e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.757249 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88cc21c0e0743999ee493f5ee856aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.758049 5142 tablet_service.cc:1467] Processing CreateTablet for tablet c565cfcf616445ff949dd75f8f513e88 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.758458 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c565cfcf616445ff949dd75f8f513e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.762639 2387 tablet_service.cc:1467] Processing CreateTablet for tablet c88cc21c0e0743999ee493f5ee856aef (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.763123 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88cc21c0e0743999ee493f5ee856aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.763527 6057 tablet_bootstrap.cc:492] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:40.765138 6057 tablet_bootstrap.cc:654] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.766999 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1f151470f24501857f38461d5b70c8 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:40.768551 6176 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:40.771497 6176 tablet.cc:1620] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:40.774247 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet fac8d3842d994279b355f9a18a82f901 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:40.774691 6147 tablet_replica.cc:330] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:40.774901 6147 raft_consensus.cc:2229] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.775177 6147 raft_consensus.cc:2258] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.776585 6147 ts_tablet_manager.cc:1875] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.786048 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 86abd8c2e86044f59c8074aa9a77db0d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.786607 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86abd8c2e86044f59c8074aa9a77db0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.787452 2006 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 68027870fbb9438fb857c1ad55311a41 (table rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd]). Expected version 1 got 0
I20240501 14:00:40.788350 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 86abd8c2e86044f59c8074aa9a77db0d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.788738 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86abd8c2e86044f59c8074aa9a77db0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.788990 4879 tablet_service.cc:1467] Processing CreateTablet for tablet c88cc21c0e0743999ee493f5ee856aef (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.789352 6147 ts_tablet_manager.cc:1888] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:40.789425 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88cc21c0e0743999ee493f5ee856aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.789471 6147 log.cc:1196] T fac8d3842d994279b355f9a18a82f901 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/fac8d3842d994279b355f9a18a82f901
I20240501 14:00:40.791164 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet fac8d3842d994279b355f9a18a82f901 (table rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]) successfully deleted
I20240501 14:00:40.791149 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 86abd8c2e86044f59c8074aa9a77db0d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:40.791580 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86abd8c2e86044f59c8074aa9a77db0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:40.803303 6177 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:40.806461 2006 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 68027870fbb9438fb857c1ad55311a41 (table rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd]). Expected version 1 got 0
I20240501 14:00:40.810905 6178 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:40.811667 6177 tablet.cc:1620] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:40.814751 6057 tablet_bootstrap.cc:492] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:40.815439 6057 ts_tablet_manager.cc:1367] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.006s
I20240501 14:00:40.816584 6057 raft_consensus.cc:348] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.816980 6057 raft_consensus.cc:374] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.817096 6057 raft_consensus.cc:729] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.817425 6057 consensus_queue.cc:260] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.818969 6183 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:40.820047 6175 tablet.cc:1620] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:40.821316 6183 tablet_bootstrap.cc:654] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.822881 6057 ts_tablet_manager.cc:1398] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:40.826334 6057 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:40.834240 6178 tablet_bootstrap.cc:654] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.835961 6180 tablet_replica.cc:330] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:40.836210 6180 raft_consensus.cc:2229] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:40.836493 6180 raft_consensus.cc:2258] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:40.837111 6181 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:40.837607 6182 tablet_bootstrap.cc:492] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:40.840015 6182 tablet_bootstrap.cc:654] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.844785 6181 tablet_bootstrap.cc:654] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.847923 6057 tablet_bootstrap.cc:654] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.848788 2001 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd] alter complete (version 1)
I20240501 14:00:40.849676 6183 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:40.849797 6180 ts_tablet_manager.cc:1875] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:40.849828 6183 ts_tablet_manager.cc:1367] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240501 14:00:40.851541 6183 raft_consensus.cc:348] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.851764 6183 raft_consensus.cc:374] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.851841 6183 raft_consensus.cc:729] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.852051 6183 consensus_queue.cc:260] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.852856 6183 ts_tablet_manager.cc:1398] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:40.852351 6182 tablet_bootstrap.cc:492] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:40.854715 6182 ts_tablet_manager.cc:1367] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240501 14:00:40.856117 6182 raft_consensus.cc:348] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.856444 6182 raft_consensus.cc:374] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.856598 6182 raft_consensus.cc:729] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.856906 6182 consensus_queue.cc:260] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.859549 6178 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:40.859723 6178 ts_tablet_manager.cc:1367] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240501 14:00:40.858654 6182 ts_tablet_manager.cc:1398] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:40.869765 6180 ts_tablet_manager.cc:1888] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:40.869957 6178 raft_consensus.cc:348] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.870286 6178 raft_consensus.cc:374] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.870409 6178 raft_consensus.cc:729] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.871400 6178 consensus_queue.cc:260] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.876448 6178 ts_tablet_manager.cc:1398] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240501 14:00:40.880476 6178 tablet_bootstrap.cc:492] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:40.882557 6057 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:40.882783 6057 ts_tablet_manager.cc:1367] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20240501 14:00:40.883569 6057 raft_consensus.cc:348] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.883847 6057 raft_consensus.cc:374] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.883960 6057 raft_consensus.cc:729] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.884250 6057 consensus_queue.cc:260] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.887009 6057 ts_tablet_manager.cc:1398] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:00:40.887373 6057 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:40.887619 6181 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:40.887835 6181 ts_tablet_manager.cc:1367] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240501 14:00:40.888749 6181 raft_consensus.cc:348] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.889047 6181 raft_consensus.cc:374] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.889191 6181 raft_consensus.cc:729] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.889536 6181 consensus_queue.cc:260] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.890259 6181 ts_tablet_manager.cc:1398] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:40.901600 6178 tablet_bootstrap.cc:654] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.904385 6057 tablet_bootstrap.cc:654] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.909902 6128 raft_consensus.cc:482] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:40.910068 6128 raft_consensus.cc:504] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.910969 6128 leader_election.cc:288] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:40.911887 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
W20240501 14:00:40.912657 4836 leader_election.cc:341] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:40.914633 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:40.914887 5164 raft_consensus.cc:2454] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:40.915468 4834 leader_election.cc:302] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:40.916015 6128 raft_consensus.cc:2781] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:40.916136 6128 raft_consensus.cc:482] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:40.916208 6128 raft_consensus.cc:3037] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:40.919772 6128 raft_consensus.cc:504] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.921010 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:40.921510 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:40.921700 5164 raft_consensus.cc:3037] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:40.922726 4836 leader_election.cc:341] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:40.925145 5164 raft_consensus.cc:2454] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:40.925446 6128 leader_election.cc:288] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:40.925714 4834 leader_election.cc:302] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:40.926391 6128 raft_consensus.cc:2781] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:40.926591 6128 raft_consensus.cc:686] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:40.927001 6128 consensus_queue.cc:237] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.929926 6178 tablet_bootstrap.cc:492] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:40.930181 6178 ts_tablet_manager.cc:1367] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240501 14:00:40.931267 6178 raft_consensus.cc:348] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.931661 6178 raft_consensus.cc:374] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.931795 6178 raft_consensus.cc:729] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.932127 6180 log.cc:1196] T 5b1f151470f24501857f38461d5b70c8 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/5b1f151470f24501857f38461d5b70c8
I20240501 14:00:40.933598 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 5b1f151470f24501857f38461d5b70c8 (table rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]) successfully deleted
I20240501 14:00:40.934540 6178 consensus_queue.cc:260] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:40.938746 6057 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:40.938948 6057 ts_tablet_manager.cc:1367] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
I20240501 14:00:40.939757 6057 raft_consensus.cc:348] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.939985 6057 raft_consensus.cc:374] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.940060 6057 raft_consensus.cc:729] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.940306 6057 consensus_queue.cc:260] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.940734 6178 ts_tablet_manager.cc:1398] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:00:40.940811 6057 ts_tablet_manager.cc:1398] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:40.944000 6178 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:40.945093 2001 catalog_manager.cc:5484] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:40.947240 6178 tablet_bootstrap.cc:654] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:40.955508 2006 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:40.965020 6175 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:40.965512 6175 tablet.cc:1620] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:40.966012 6177 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:40.966485 6177 tablet.cc:1620] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:40.966667 6176 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:40.967144 6176 tablet.cc:1620] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:40.981101 2006 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=60c25958e7fc41caa3b6708a90f5c8dd] alter complete (version 2)
I20240501 14:00:40.990710 6178 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:40.990932 6178 ts_tablet_manager.cc:1367] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.003s
I20240501 14:00:40.991734 6178 raft_consensus.cc:348] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.991979 6178 raft_consensus.cc:374] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:40.992062 6178 raft_consensus.cc:729] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:40.992285 6178 consensus_queue.cc:260] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:40.992815 6178 ts_tablet_manager.cc:1398] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:41.002741 6190 raft_consensus.cc:482] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:41.002929 6190 raft_consensus.cc:504] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:41.003710 6190 leader_election.cc:288] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.004245 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86abd8c2e86044f59c8074aa9a77db0d" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:41.004957 5293 raft_consensus.cc:2454] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:41.005270 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86abd8c2e86044f59c8074aa9a77db0d" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:41.005455 4836 leader_election.cc:302] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:41.005534 5033 raft_consensus.cc:2454] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:41.005853 6190 raft_consensus.cc:2781] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:41.005976 6190 raft_consensus.cc:482] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:41.006070 6190 raft_consensus.cc:3037] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.008248 6190 raft_consensus.cc:504] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:41.008951 6190 leader_election.cc:288] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.010228 5033 consensus_queue.cc:237] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:00:41.010341 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86abd8c2e86044f59c8074aa9a77db0d" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:41.010586 5293 raft_consensus.cc:3037] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.011155 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86abd8c2e86044f59c8074aa9a77db0d" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:41.011346 5032 raft_consensus.cc:3037] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.013338 5293 raft_consensus.cc:2454] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:41.014010 5032 raft_consensus.cc:2454] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:41.017365 4901 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.018015 4836 leader_election.cc:302] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:41.019215 5973 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.019483 6128 raft_consensus.cc:2781] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:41.019627 6128 raft_consensus.cc:686] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:41.019871 6128 consensus_queue.cc:237] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:41.023562 2006 catalog_manager.cc:5484] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:41.025712 5973 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:00:41.032259 4901 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:00:41.040581 2006 catalog_manager.cc:5484] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:00:41.048962 5997 raft_consensus.cc:482] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:41.049177 5997 raft_consensus.cc:504] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:41.049909 5997 leader_election.cc:288] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:41.051096 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cc21c0e0743999ee493f5ee856aef" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:41.051383 2406 raft_consensus.cc:2454] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:41.052042 5228 leader_election.cc:302] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:41.052443 5997 raft_consensus.cc:2781] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:41.052551 5997 raft_consensus.cc:482] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:41.052645 5997 raft_consensus.cc:3037] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:41.054724 4967 consensus_peers.cc:485] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c685294d3fca4677bc1b1cf355dac3ce. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:41.055328 5997 raft_consensus.cc:504] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:41.056520 2147 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.056632 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cc21c0e0743999ee493f5ee856aef" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:41.056864 2406 raft_consensus.cc:3037] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.059747 2406 raft_consensus.cc:2454] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:41.060523 5228 leader_election.cc:302] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:41.060906 5796 raft_consensus.cc:2781] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:41.061046 5796 raft_consensus.cc:686] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:41.061313 5796 consensus_queue.cc:237] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:41.061054 6208 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 14:00:41.066314 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cc21c0e0743999ee493f5ee856aef" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:41.066666 4901 raft_consensus.cc:2454] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:41.070034 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88cc21c0e0743999ee493f5ee856aef" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:41.070328 4901 raft_consensus.cc:3037] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.072243 5997 leader_election.cc:288] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:41.075037 4901 raft_consensus.cc:2454] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:41.088552 2006 catalog_manager.cc:5484] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:41.091624 2001 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:41.102737 6175 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:41.104530 6176 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:41.107596 6177 tablet.cc:1722] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:41.111622 5033 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:41.117987 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=60c25958e7fc41caa3b6708a90f5c8dd] alter complete (version 3)
I20240501 14:00:41.122568 2147 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
W20240501 14:00:41.129854 4967 consensus_peers.cc:485] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf051cecef424db5accfce66623346b5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:41.130111 4901 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.131004 6208 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.131794 5164 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.132900 6208 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.136823 5994 raft_consensus.cc:482] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:41.137018 5994 raft_consensus.cc:504] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.137677 5994 leader_election.cc:288] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:41.138412 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:41.139597 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:41.139868 5163 raft_consensus.cc:2379] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a4b9a96e9734baa8a03d6056bfa934c in current term 1: Already voted for candidate f8c60f2bbac541dcba4424ab726f9ad9 in this term.
I20240501 14:00:41.140420 5228 leader_election.cc:302] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:41.141156 5994 raft_consensus.cc:3037] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.144055 5994 raft_consensus.cc:2726] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:41.147506 5973 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:41.148488 5164 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:41.150166 4901 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:41.159294 2006 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:41.185588 2535 consensus_queue.cc:237] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
W20240501 14:00:41.195896 2472 consensus_peers.cc:485] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb1fa60c64034b7d8c7b4879823b0d83. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:41.195899 5293 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.196455 5727 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.197600 5164 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.198146 5727 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.199946 5727 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:41.203433 5293 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:41.203785 5164 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:41.209226 2006 catalog_manager.cc:5484] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:41.224041 2406 consensus_queue.cc:237] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:41.226272 2275 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:41.226359 2535 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:41.226919 2343 consensus_peers.cc:485] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c74bacbf4464bab9c403e2153953474. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:41.226966 6132 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:41.227331 5713 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:41.230000 2001 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:41.230228 2001 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:41.230147 4800 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:41.231112 2535 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:41.233753 2275 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:41.236176 2006 catalog_manager.cc:5484] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:41.238849 2001 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 68027870fbb9438fb857c1ad55311a41
I20240501 14:00:41.239616 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet 68027870fbb9438fb857c1ad55311a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:41 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:41.239989 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 68027870fbb9438fb857c1ad55311a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:41 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:41.240331 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet 68027870fbb9438fb857c1ad55311a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:41 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:41.240721 6180 tablet_replica.cc:330] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:41.240933 6180 raft_consensus.cc:2229] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:41.241343 6180 raft_consensus.cc:2258] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:41.242866 6180 ts_tablet_manager.cc:1875] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:41.245844 6239 tablet_replica.cc:330] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:41.246112 6239 raft_consensus.cc:2229] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:41.246346 6239 raft_consensus.cc:2258] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:41.248025 6239 ts_tablet_manager.cc:1875] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:41.250633 6180 ts_tablet_manager.cc:1888] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:41.250888 6180 log.cc:1196] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/68027870fbb9438fb857c1ad55311a41
I20240501 14:00:41.251454 6180 ts_tablet_manager.cc:1909] T 68027870fbb9438fb857c1ad55311a41 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:41.252864 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 68027870fbb9438fb857c1ad55311a41 (table rebalancer_dynamic_table_new_name [id=60c25958e7fc41caa3b6708a90f5c8dd]) successfully deleted
I20240501 14:00:41.254014 6238 tablet_replica.cc:330] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:41.254225 6238 raft_consensus.cc:2229] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:41.254490 6238 raft_consensus.cc:2258] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:41.255371 6238 ts_tablet_manager.cc:1875] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:41.256060 6239 ts_tablet_manager.cc:1888] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:41.256177 6239 log.cc:1196] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/68027870fbb9438fb857c1ad55311a41
I20240501 14:00:41.256583 6239 ts_tablet_manager.cc:1909] T 68027870fbb9438fb857c1ad55311a41 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:41.257606 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 68027870fbb9438fb857c1ad55311a41 (table rebalancer_dynamic_table_new_name [id=60c25958e7fc41caa3b6708a90f5c8dd]) successfully deleted
I20240501 14:00:41.260160 6238 ts_tablet_manager.cc:1888] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:41.260274 6238 log.cc:1196] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/68027870fbb9438fb857c1ad55311a41
I20240501 14:00:41.260814 6238 ts_tablet_manager.cc:1909] T 68027870fbb9438fb857c1ad55311a41 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:41.262220 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 68027870fbb9438fb857c1ad55311a41 (table rebalancer_dynamic_table_new_name [id=60c25958e7fc41caa3b6708a90f5c8dd]) successfully deleted
I20240501 14:00:41.276309 2406 consensus_queue.cc:237] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
W20240501 14:00:41.279574 2341 consensus_peers.cc:485] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9d8d993ce834c1db4dd4418f7a85780. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:41.279700 2147 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:41.279937 2275 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:41.280555 6132 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:41.281013 4800 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:41.284651 5713 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:41.285367 2275 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:41.285373 2147 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:41.291348 2001 catalog_manager.cc:5484] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:41.309767 5293 consensus_queue.cc:237] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:00:41.312309 4901 raft_consensus.cc:1261] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.312309 5164 raft_consensus.cc:1261] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:00:41.312583 5228 consensus_peers.cc:485] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9b8c92907e843409e37919a1a0e96af. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:41.313052 5997 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.313388 6213 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.316007 5994 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:00:41.316742 5164 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:00:41.316711 4901 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:00:41.322806 1999 catalog_manager.cc:5484] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:41.339474 2275 consensus_queue.cc:237] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:41.341643 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240501 14:00:41.342347 2211 consensus_peers.cc:485] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9935f0f87637467091ff85989503af28. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:41.342842 4901 raft_consensus.cc:1261] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.343449 6027 consensus_queue.cc:1035] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.344043 5293 raft_consensus.cc:1261] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.345180 6027 consensus_queue.cc:1035] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:41.351994 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2c401596ee460784920a2c68477902 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3]), partition=
I20240501 14:00:41.352249 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2c401596ee460784920a2c68477902 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3]), partition=
I20240501 14:00:41.352387 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2c401596ee460784920a2c68477902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.352622 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2c401596ee460784920a2c68477902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.353310 6027 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:41.355960 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 3a2c401596ee460784920a2c68477902 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3]), partition=
I20240501 14:00:41.356330 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a2c401596ee460784920a2c68477902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.361770 6057 tablet_bootstrap.cc:492] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:41.361694 4901 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:41.363507 6057 tablet_bootstrap.cc:654] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.368477 2003 catalog_manager.cc:5484] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:41.372902 5296 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:41.378760 6178 tablet_bootstrap.cc:492] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:41.381029 6178 tablet_bootstrap.cc:654] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.383332 6178 tablet_bootstrap.cc:492] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:41.383584 6178 ts_tablet_manager.cc:1367] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:41.384317 6178 raft_consensus.cc:348] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.384861 6178 raft_consensus.cc:374] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.385001 6178 raft_consensus.cc:729] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.385311 6178 consensus_queue.cc:260] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.385931 6178 ts_tablet_manager.cc:1398] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:41.398358 6057 tablet_bootstrap.cc:492] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:41.398599 6057 ts_tablet_manager.cc:1367] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20240501 14:00:41.400383 6190 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.401015 6057 raft_consensus.cc:348] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.401226 6057 raft_consensus.cc:374] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.401319 6057 raft_consensus.cc:729] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.401715 6057 consensus_queue.cc:260] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.402282 6057 ts_tablet_manager.cc:1398] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:00:41.406944 6190 raft_consensus.cc:482] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:41.407122 6190 raft_consensus.cc:504] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.407811 6190 leader_election.cc:288] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.408520 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2c401596ee460784920a2c68477902" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:41.408955 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2c401596ee460784920a2c68477902" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:41.409166 5293 raft_consensus.cc:2454] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:41.409250 6272 ts_tablet_manager.cc:902] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:41.409641 4836 leader_election.cc:302] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:41.410022 6190 raft_consensus.cc:2781] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:41.410116 6190 raft_consensus.cc:482] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:41.410228 6190 raft_consensus.cc:3037] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.411522 6270 tablet_bootstrap.cc:492] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:41.412941 6190 raft_consensus.cc:504] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.414151 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2c401596ee460784920a2c68477902" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:41.414938 6270 tablet_bootstrap.cc:654] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:41.416776 4834 leader_election.cc:341] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:41.419400 6190 leader_election.cc:288] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.420502 6190 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.422571 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a2c401596ee460784920a2c68477902" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:41.422808 5293 raft_consensus.cc:3037] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.422868 6270 tablet_bootstrap.cc:492] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:41.423027 6270 ts_tablet_manager.cc:1367] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:00:41.424156 6270 raft_consensus.cc:348] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.424364 6270 raft_consensus.cc:374] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.424445 6270 raft_consensus.cc:729] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.424641 6270 consensus_queue.cc:260] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.425042 6270 ts_tablet_manager.cc:1398] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:41.425827 5293 raft_consensus.cc:2454] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:41.426990 4836 leader_election.cc:302] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters: d1a414d381f44829a2510cb76d481887
I20240501 14:00:41.427598 6191 raft_consensus.cc:2781] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:41.427747 6191 raft_consensus.cc:686] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:41.428050 6191 consensus_queue.cc:237] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.452080 1999 catalog_manager.cc:5484] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:41.452381 5994 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.459858 6272 tablet_copy_client.cc:308] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.6:46037
I20240501 14:00:41.461259 5054 tablet_copy_service.cc:140] P 6f81f803ec7746ccb502c59b7cbf8eb2: Received BeginTabletCopySession request for tablet c685294d3fca4677bc1b1cf355dac3ce from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:45021)
I20240501 14:00:41.461491 5054 tablet_copy_service.cc:161] P 6f81f803ec7746ccb502c59b7cbf8eb2: Beginning new tablet copy session on tablet c685294d3fca4677bc1b1cf355dac3ce from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:45021: session id = 745ac12076e84effa5bde1ecbd6a32ec-c685294d3fca4677bc1b1cf355dac3ce
I20240501 14:00:41.464711 5054 tablet_copy_source_session.cc:205] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:41.467016 6213 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.469583 6272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c685294d3fca4677bc1b1cf355dac3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.478293 6272 tablet_copy_client.cc:783] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:41.478907 6272 tablet_copy_client.cc:647] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:41.481299 6272 tablet_copy_client.cc:523] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:41.485077 6272 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:41.487437 6191 consensus_queue.cc:1035] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.499425 6272 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:41.500149 6272 tablet_bootstrap.cc:492] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:00:41.500524 6272 ts_tablet_manager.cc:1367] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20240501 14:00:41.503369 6272 raft_consensus.cc:348] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:00:41.503793 6272 raft_consensus.cc:729] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:00:41.504063 6272 consensus_queue.cc:260] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:00:41.504784 6272 ts_tablet_manager.cc:1398] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:41.505770 5054 tablet_copy_service.cc:342] P 6f81f803ec7746ccb502c59b7cbf8eb2: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-c685294d3fca4677bc1b1cf355dac3ce received from {username='slave'} at 127.28.17.4:45021
I20240501 14:00:41.505964 5054 tablet_copy_service.cc:433] P 6f81f803ec7746ccb502c59b7cbf8eb2: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-c685294d3fca4677bc1b1cf355dac3ce on tablet c685294d3fca4677bc1b1cf355dac3ce with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:00:41.514627 6191 consensus_queue.cc:1035] T c565cfcf616445ff949dd75f8f513e88 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.571725 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:41.577371 2407 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84"
dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41864
I20240501 14:00:41.577687 2407 raft_consensus.cc:595] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Received request to transfer leadership
I20240501 14:00:41.577898 6327 ts_tablet_manager.cc:902] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:41.578271 5296 raft_consensus.cc:1261] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.578445 2147 raft_consensus.cc:3037] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.579166 6191 consensus_queue.cc:1035] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.580968 2147 raft_consensus.cc:1261] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:41.581665 6191 consensus_queue.cc:1035] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:41.587092 6177 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:41.587705 6177 tablet.cc:1620] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:41.596058 6176 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:41.595863 6336 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:41.596609 6176 tablet.cc:1620] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:41.597061 6336 tablet.cc:1620] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:00:41.603850 1999 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 3a2c401596ee460784920a2c68477902 (table rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3]). Expected version 1 got 0
I20240501 14:00:41.607440 2003 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 3a2c401596ee460784920a2c68477902 (table rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3]). Expected version 1 got 0
I20240501 14:00:41.608076 2006 catalog_manager.cc:5576] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309) does not have the latest schema for tablet 3a2c401596ee460784920a2c68477902 (table rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3]). Expected version 1 got 0
I20240501 14:00:41.615628 2006 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3] alter complete (version 1)
I20240501 14:00:41.622830 6327 tablet_copy_client.cc:308] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.6:46037
I20240501 14:00:41.623492 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:41.623801 5054 tablet_copy_service.cc:140] P 6f81f803ec7746ccb502c59b7cbf8eb2: Received BeginTabletCopySession request for tablet bf051cecef424db5accfce66623346b5 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:48677)
I20240501 14:00:41.623991 5054 tablet_copy_service.cc:161] P 6f81f803ec7746ccb502c59b7cbf8eb2: Beginning new tablet copy session on tablet bf051cecef424db5accfce66623346b5 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:48677: session id = a64f34785a7b44fa8070b0452a4e0ddd-bf051cecef424db5accfce66623346b5
I20240501 14:00:41.626466 5054 tablet_copy_source_session.cc:205] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:41.628630 6327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf051cecef424db5accfce66623346b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.636607 6327 tablet_copy_client.cc:783] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:41.637066 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 27abced2b6b746d2b7c45883a328a3d3 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.637379 6327 tablet_copy_client.cc:647] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:41.637537 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27abced2b6b746d2b7c45883a328a3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.639137 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdc95819b944c5590202e86cc955d91 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.639606 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdc95819b944c5590202e86cc955d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.640291 6327 tablet_copy_client.cc:523] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:41.643249 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdc95819b944c5590202e86cc955d91 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.643724 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdc95819b944c5590202e86cc955d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.645454 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 27abced2b6b746d2b7c45883a328a3d3 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.645776 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdc95819b944c5590202e86cc955d91 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.645875 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27abced2b6b746d2b7c45883a328a3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.646209 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdc95819b944c5590202e86cc955d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.650936 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 27abced2b6b746d2b7c45883a328a3d3 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.651355 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27abced2b6b746d2b7c45883a328a3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.653421 6327 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:41.656597 2255 tablet_service.cc:1467] Processing CreateTablet for tablet a7b11568c2d7420fa06f903e52723583 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.657073 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b11568c2d7420fa06f903e52723583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.657006 5013 tablet_service.cc:1467] Processing CreateTablet for tablet a7b11568c2d7420fa06f903e52723583 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.657414 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b11568c2d7420fa06f903e52723583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.659456 4881 tablet_service.cc:1467] Processing CreateTablet for tablet a7b11568c2d7420fa06f903e52723583 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:41.659873 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b11568c2d7420fa06f903e52723583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.666157 6057 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:41.668004 6057 tablet_bootstrap.cc:654] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.669924 6178 tablet_bootstrap.cc:492] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:41.672236 6178 tablet_bootstrap.cc:654] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.681389 6270 tablet_bootstrap.cc:492] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:41.683128 6270 tablet_bootstrap.cc:654] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.683182 6346 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:41.688395 6350 ts_tablet_manager.cc:902] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:41.692031 6347 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:41.694713 6346 tablet_bootstrap.cc:654] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.696116 6347 tablet_bootstrap.cc:654] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.700599 6270 tablet_bootstrap.cc:492] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:41.701018 6270 ts_tablet_manager.cc:1367] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240501 14:00:41.702008 6270 raft_consensus.cc:348] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.704355 6270 raft_consensus.cc:374] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.704526 6270 raft_consensus.cc:729] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.704905 6270 consensus_queue.cc:260] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.708187 6270 ts_tablet_manager.cc:1398] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:00:41.716930 6351 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:41.719511 6351 tablet_bootstrap.cc:654] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.722422 6178 tablet_bootstrap.cc:492] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:41.721845 6346 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:41.722738 6178 ts_tablet_manager.cc:1367] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.010s
I20240501 14:00:41.722810 6346 ts_tablet_manager.cc:1367] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.000s
I20240501 14:00:41.722987 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:41.724071 6178 raft_consensus.cc:348] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.724406 6178 raft_consensus.cc:374] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.724553 6178 raft_consensus.cc:729] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.724884 6178 consensus_queue.cc:260] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.726892 6178 ts_tablet_manager.cc:1398] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:41.728165 6057 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:41.728581 6057 ts_tablet_manager.cc:1367] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.006s
I20240501 14:00:41.729667 6057 raft_consensus.cc:348] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.730016 6057 raft_consensus.cc:374] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.730170 6057 raft_consensus.cc:729] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.730551 6057 consensus_queue.cc:260] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.730695 6062 raft_consensus.cc:482] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:41.730854 6062 raft_consensus.cc:504] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.731702 6062 leader_election.cc:288] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.732117 6057 ts_tablet_manager.cc:1398] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:41.732427 6057 tablet_bootstrap.cc:492] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:41.735899 4800 raft_consensus.cc:979] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: : Instructing follower d1a414d381f44829a2510cb76d481887 to start an election
I20240501 14:00:41.736111 4799 raft_consensus.cc:1067] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Signalling peer d1a414d381f44829a2510cb76d481887 to start an election
I20240501 14:00:41.731251 6346 raft_consensus.cc:348] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.736828 6346 raft_consensus.cc:374] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.736945 6346 raft_consensus.cc:729] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.737622 6346 consensus_queue.cc:260] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.739040 2147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84"
dest_uuid: "d1a414d381f44829a2510cb76d481887"
from {username='slave'} at 127.28.17.3:50193
I20240501 14:00:41.739313 2147 raft_consensus.cc:482] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:41.739445 2147 raft_consensus.cc:3037] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:41.743685 6346 ts_tablet_manager.cc:1398] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:00:41.742477 2147 raft_consensus.cc:504] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:41.746148 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:41.746392 2275 raft_consensus.cc:3037] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:41.749353 2275 raft_consensus.cc:2454] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:00:41.751019 6360 ts_tablet_manager.cc:902] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:41.751025 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:41.751238 2406 raft_consensus.cc:3032] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:41.751344 2406 raft_consensus.cc:729] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:41.751672 2406 consensus_queue.cc:260] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:41.752135 2406 raft_consensus.cc:3037] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:41.755915 2406 raft_consensus.cc:2454] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:00:41.756549 2080 leader_election.cc:302] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:41.756966 6062 raft_consensus.cc:2781] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:41.758170 2147 leader_election.cc:288] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:41.760015 6057 tablet_bootstrap.cc:654] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.764668 6347 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:41.764858 6347 ts_tablet_manager.cc:1367] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.000s
I20240501 14:00:41.765684 6347 raft_consensus.cc:348] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.765900 6347 raft_consensus.cc:374] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.765996 6347 raft_consensus.cc:729] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.766242 6347 consensus_queue.cc:260] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.779366 6350 tablet_copy_client.cc:308] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:41.780392 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet b9d8d993ce834c1db4dd4418f7a85780 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:46539)
I20240501 14:00:41.780572 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet b9d8d993ce834c1db4dd4418f7a85780 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:46539: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-b9d8d993ce834c1db4dd4418f7a85780
I20240501 14:00:41.781592 6347 ts_tablet_manager.cc:1398] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:00:41.781587 6177 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:41.782117 6177 tablet.cc:1620] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:41.782704 6272 ts_tablet_manager.cc:902] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.784497 2428 tablet_copy_source_session.cc:205] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:41.786870 6327 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:41.787523 6327 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:00:41.787809 6327 ts_tablet_manager.cc:1367] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.136s user 0.007s sys 0.005s
I20240501 14:00:41.788033 6347 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:41.790472 6327 raft_consensus.cc:348] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:41.790776 6327 raft_consensus.cc:729] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:00:41.791038 6327 consensus_queue.cc:260] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:41.733422 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27abced2b6b746d2b7c45883a328a3d3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:41.791596 5296 raft_consensus.cc:2454] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:41.786926 6176 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:41.792249 6176 tablet.cc:1620] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:41.792964 2082 leader_election.cc:302] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:41.796497 6336 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:41.796914 6057 tablet_bootstrap.cc:492] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:41.796964 6336 tablet.cc:1620] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:00:41.797034 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=74c89cc4ee6241f0b2066ff5cb2822b3] alter complete (version 2)
I20240501 14:00:41.798202 6350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8d993ce834c1db4dd4418f7a85780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.804344 6347 tablet_bootstrap.cc:654] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.815455 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27abced2b6b746d2b7c45883a328a3d3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:41.816426 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b7e8e96f4e4cc0bd2c824882ac4d84" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:41.816653 4901 raft_consensus.cc:3037] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:41.818150 6377 raft_consensus.cc:2781] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:41.818270 6377 raft_consensus.cc:482] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:41.818369 6377 raft_consensus.cc:3037] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.820155 4901 raft_consensus.cc:2454] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
W20240501 14:00:41.821066 2081 leader_election.cc:341] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:41.821910 6062 raft_consensus.cc:686] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:41.822171 6057 ts_tablet_manager.cc:1367] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.003s
I20240501 14:00:41.822448 6062 consensus_queue.cc:237] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:41.823451 6057 raft_consensus.cc:348] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.825250 6057 raft_consensus.cc:374] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.825623 6178 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:41.825654 6057 raft_consensus.cc:729] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.826210 6057 consensus_queue.cc:260] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.828136 6178 tablet_bootstrap.cc:654] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:41.829898 2534 raft_consensus.cc:1203] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:41.834594 6057 ts_tablet_manager.cc:1398] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 14:00:41.842789 6351 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:41.843088 6351 ts_tablet_manager.cc:1367] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.000s
I20240501 14:00:41.843938 6351 raft_consensus.cc:348] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.844805 6351 raft_consensus.cc:374] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.844954 6351 raft_consensus.cc:729] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.845250 6351 consensus_queue.cc:260] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.845877 6351 ts_tablet_manager.cc:1398] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:41.846071 6327 ts_tablet_manager.cc:1398] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.058s user 0.001s sys 0.002s
I20240501 14:00:41.851913 6377 raft_consensus.cc:504] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.857833 6360 tablet_copy_client.cc:308] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:41.853102 5054 tablet_copy_service.cc:342] P 6f81f803ec7746ccb502c59b7cbf8eb2: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-bf051cecef424db5accfce66623346b5 received from {username='slave'} at 127.28.17.3:48677
I20240501 14:00:41.858237 5054 tablet_copy_service.cc:433] P 6f81f803ec7746ccb502c59b7cbf8eb2: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-bf051cecef424db5accfce66623346b5 on tablet bf051cecef424db5accfce66623346b5 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:41.858667 6377 leader_election.cc:288] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.858891 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet eb1fa60c64034b7d8c7b4879823b0d83 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:50935)
I20240501 14:00:41.859091 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet eb1fa60c64034b7d8c7b4879823b0d83 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:50935: session id = f8c60f2bbac541dcba4424ab726f9ad9-eb1fa60c64034b7d8c7b4879823b0d83
I20240501 14:00:41.859201 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27abced2b6b746d2b7c45883a328a3d3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:41.859436 5296 raft_consensus.cc:3037] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.859845 6178 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:41.861377 6327 ts_tablet_manager.cc:902] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:41.862708 2558 tablet_copy_source_session.cc:205] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:41.862886 5296 raft_consensus.cc:2454] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:41.860026 6178 ts_tablet_manager.cc:1367] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240501 14:00:41.863422 2082 leader_election.cc:302] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:41.864080 6178 raft_consensus.cc:348] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.864387 6178 raft_consensus.cc:374] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.864282 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27abced2b6b746d2b7c45883a328a3d3" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:41.864547 4901 raft_consensus.cc:3037] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.864671 6178 raft_consensus.cc:729] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.864591 6377 raft_consensus.cc:2781] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:41.865111 6178 consensus_queue.cc:260] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.865017 2006 catalog_manager.cc:5484] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 1 to 2, leader changed from a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:41.865449 6377 raft_consensus.cc:686] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:41.865900 6377 consensus_queue.cc:237] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.867566 4901 raft_consensus.cc:2454] T 27abced2b6b746d2b7c45883a328a3d3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:41.868760 6178 ts_tablet_manager.cc:1398] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:41.869638 6360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1fa60c64034b7d8c7b4879823b0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.874270 6350 tablet_copy_client.cc:783] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:41.875274 6350 tablet_copy_client.cc:647] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:41.877336 6350 tablet_copy_client.cc:523] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:41.881084 2006 catalog_manager.cc:5484] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:41.882361 6347 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:41.883740 6347 ts_tablet_manager.cc:1367] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.018s
I20240501 14:00:41.884574 6350 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:41.884577 6347 raft_consensus.cc:348] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.885211 6347 raft_consensus.cc:374] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:41.885625 6347 raft_consensus.cc:729] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:41.886096 6347 consensus_queue.cc:260] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.895790 6387 ts_tablet_manager.cc:902] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:41.899252 6347 ts_tablet_manager.cc:1398] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:00:41.900362 2006 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:41.903697 5965 raft_consensus.cc:482] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:41.903887 5965 raft_consensus.cc:504] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.906608 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdc95819b944c5590202e86cc955d91" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:41.906880 5296 raft_consensus.cc:2454] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:41.910006 5099 leader_election.cc:302] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:41.909556 5965 leader_election.cc:288] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.910786 5965 raft_consensus.cc:2781] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:41.910916 5965 raft_consensus.cc:482] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:41.910990 5965 raft_consensus.cc:3037] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.917064 6177 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:41.920224 6176 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:41.923861 6336 tablet.cc:1722] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:41.937882 2006 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=74c89cc4ee6241f0b2066ff5cb2822b3] alter complete (version 3)
I20240501 14:00:41.945762 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdc95819b944c5590202e86cc955d91" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:41.946022 2276 raft_consensus.cc:2454] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:41.948506 6327 tablet_copy_client.cc:308] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:00:41.949415 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 9935f0f87637467091ff85989503af28 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:44257)
I20240501 14:00:41.949923 6360 tablet_copy_client.cc:783] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:41.949908 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 9935f0f87637467091ff85989503af28 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:44257: session id = a64f34785a7b44fa8070b0452a4e0ddd-9935f0f87637467091ff85989503af28
I20240501 14:00:41.950002 6387 tablet_copy_client.cc:308] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:41.950273 6360 tablet_copy_client.cc:647] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:41.951545 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet 6c74bacbf4464bab9c403e2153953474 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:41439)
I20240501 14:00:41.951711 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet 6c74bacbf4464bab9c403e2153953474 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:41439: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-6c74bacbf4464bab9c403e2153953474
I20240501 14:00:41.952479 6360 tablet_copy_client.cc:523] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:41.954012 2298 tablet_copy_source_session.cc:205] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:41.955001 6354 raft_consensus.cc:482] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:41.955175 6354 raft_consensus.cc:504] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.955518 2428 tablet_copy_source_session.cc:205] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:41.955976 6354 leader_election.cc:288] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:41.956715 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b11568c2d7420fa06f903e52723583" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:41.956936 4901 raft_consensus.cc:2454] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:41.957392 4968 leader_election.cc:302] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:41.957680 6354 raft_consensus.cc:2781] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:41.957789 6354 raft_consensus.cc:482] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:41.957877 6354 raft_consensus.cc:3037] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.960884 6354 raft_consensus.cc:504] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.962870 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b11568c2d7420fa06f903e52723583" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:41.963145 4901 raft_consensus.cc:3037] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.964352 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c74bacbf4464bab9c403e2153953474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.966010 4901 raft_consensus.cc:2454] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:41.966764 4968 leader_election.cc:302] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:41.967092 6362 raft_consensus.cc:2781] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:41.967231 6362 raft_consensus.cc:686] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:41.967540 6362 consensus_queue.cc:237] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:41.968230 6327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935f0f87637467091ff85989503af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:41.975489 6360 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:41.975713 6327 tablet_copy_client.cc:783] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:41.975944 6327 tablet_copy_client.cc:647] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:41.978399 6327 tablet_copy_client.cc:523] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:41.979674 1999 catalog_manager.cc:5484] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:41.980340 6354 leader_election.cc:288] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:41.980809 5965 raft_consensus.cc:504] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.981974 6327 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:41.982908 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdc95819b944c5590202e86cc955d91" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:41.983242 2276 raft_consensus.cc:3037] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.984920 6387 tablet_copy_client.cc:783] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:41.985267 6387 tablet_copy_client.cc:647] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:41.986785 2276 raft_consensus.cc:2454] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:41.987334 6387 tablet_copy_client.cc:523] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:41.987413 5097 leader_election.cc:302] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:41.988145 5965 leader_election.cc:288] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:41.988905 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdc95819b944c5590202e86cc955d91" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:41.989302 5296 raft_consensus.cc:3037] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.990734 5965 raft_consensus.cc:2781] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:41.991029 5965 raft_consensus.cc:686] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:41.991057 6387 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:41.991425 5965 consensus_queue.cc:237] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:41.993463 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b11568c2d7420fa06f903e52723583" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:41.993685 2275 raft_consensus.cc:3037] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:41.993860 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b11568c2d7420fa06f903e52723583" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:41.996997 2275 raft_consensus.cc:2454] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:42.001221 5296 raft_consensus.cc:2454] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:42.009523 6360 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:42.010722 6360 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:42.011111 6360 ts_tablet_manager.cc:1367] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.000s
I20240501 14:00:42.012111 6360 raft_consensus.cc:348] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:42.012364 6360 raft_consensus.cc:729] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:42.012619 6360 consensus_queue.cc:260] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:42.014290 6360 ts_tablet_manager.cc:1398] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:42.015141 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-eb1fa60c64034b7d8c7b4879823b0d83 received from {username='slave'} at 127.28.17.5:50935
I20240501 14:00:42.015302 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-eb1fa60c64034b7d8c7b4879823b0d83 on tablet eb1fa60c64034b7d8c7b4879823b0d83 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:42.023105 1999 catalog_manager.cc:5484] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:42.029675 6327 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:42.030257 6327 tablet_bootstrap.cc:492] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:00:42.030685 6327 ts_tablet_manager.cc:1367] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.012s
I20240501 14:00:42.032253 6327 raft_consensus.cc:348] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:42.033283 6327 raft_consensus.cc:729] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:00:42.033632 6327 consensus_queue.cc:260] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:42.034457 6327 ts_tablet_manager.cc:1398] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:42.037662 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-9935f0f87637467091ff85989503af28 received from {username='slave'} at 127.28.17.3:44257
I20240501 14:00:42.037859 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-9935f0f87637467091ff85989503af28 on tablet 9935f0f87637467091ff85989503af28 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:42.041879 1999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:42.042253 1999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:42.046005 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 3a2c401596ee460784920a2c68477902
I20240501 14:00:42.047150 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2c401596ee460784920a2c68477902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:42 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:42.047235 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2c401596ee460784920a2c68477902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:42 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:42.047564 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a2c401596ee460784920a2c68477902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:42 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:42.057508 6272 tablet_copy_client.cc:308] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:00:42.058785 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet f9b8c92907e843409e37919a1a0e96af from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:35953)
I20240501 14:00:42.059051 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet f9b8c92907e843409e37919a1a0e96af from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:35953: session id = 745ac12076e84effa5bde1ecbd6a32ec-f9b8c92907e843409e37919a1a0e96af
I20240501 14:00:42.063781 5316 tablet_copy_source_session.cc:205] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:42.066176 6272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b8c92907e843409e37919a1a0e96af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.071780 6387 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:42.072391 6387 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:42.072891 6387 ts_tablet_manager.cc:1367] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.082s user 0.049s sys 0.000s
I20240501 14:00:42.073851 6387 raft_consensus.cc:348] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:42.074352 6387 raft_consensus.cc:729] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:42.075603 6420 tablet_replica.cc:330] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:42.075822 6420 raft_consensus.cc:2229] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:42.076212 6420 raft_consensus.cc:2258] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:42.078015 6418 tablet_replica.cc:330] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:42.078415 6418 raft_consensus.cc:2229] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:42.078773 6418 raft_consensus.cc:2258] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:42.079586 6387 consensus_queue.cc:260] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:42.079864 6418 ts_tablet_manager.cc:1875] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:42.081053 6419 tablet_replica.cc:330] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:42.081264 6419 raft_consensus.cc:2229] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:42.082751 6419 raft_consensus.cc:2258] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:42.083498 6420 ts_tablet_manager.cc:1875] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:42.085044 6387 ts_tablet_manager.cc:1398] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:00:42.085994 6418 ts_tablet_manager.cc:1888] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:42.087215 6419 ts_tablet_manager.cc:1875] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:42.090703 6418 log.cc:1196] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a2c401596ee460784920a2c68477902
I20240501 14:00:42.090859 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-6c74bacbf4464bab9c403e2153953474 received from {username='slave'} at 127.28.17.6:41439
I20240501 14:00:42.090996 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-6c74bacbf4464bab9c403e2153953474 on tablet 6c74bacbf4464bab9c403e2153953474 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:42.091603 6418 ts_tablet_manager.cc:1909] T 3a2c401596ee460784920a2c68477902 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:42.093652 6272 tablet_copy_client.cc:783] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:42.093919 6272 tablet_copy_client.cc:647] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:42.093982 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 3a2c401596ee460784920a2c68477902 (table rebalancer_dynamic_table_new_name [id=74c89cc4ee6241f0b2066ff5cb2822b3]) successfully deleted
I20240501 14:00:42.102128 6272 tablet_copy_client.cc:523] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:42.104154 6419 ts_tablet_manager.cc:1888] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:42.104312 6419 log.cc:1196] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/3a2c401596ee460784920a2c68477902
I20240501 14:00:42.105979 6272 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:42.110075 6420 ts_tablet_manager.cc:1888] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:42.115140 6419 ts_tablet_manager.cc:1909] T 3a2c401596ee460784920a2c68477902 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:42.115366 6420 log.cc:1196] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/3a2c401596ee460784920a2c68477902
I20240501 14:00:42.115872 6420 ts_tablet_manager.cc:1909] T 3a2c401596ee460784920a2c68477902 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:42.117012 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 3a2c401596ee460784920a2c68477902 (table rebalancer_dynamic_table_new_name [id=74c89cc4ee6241f0b2066ff5cb2822b3]) successfully deleted
I20240501 14:00:42.118328 6272 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:42.118862 6272 tablet_bootstrap.cc:492] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:00:42.119181 6272 ts_tablet_manager.cc:1367] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240501 14:00:42.120157 6272 raft_consensus.cc:348] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:00:42.120471 6272 raft_consensus.cc:729] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:00:42.120715 6272 consensus_queue.cc:260] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:00:42.122551 6272 ts_tablet_manager.cc:1398] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:42.124318 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-f9b8c92907e843409e37919a1a0e96af received from {username='slave'} at 127.28.17.4:35953
I20240501 14:00:42.124511 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-f9b8c92907e843409e37919a1a0e96af on tablet f9b8c92907e843409e37919a1a0e96af with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:00:42.130649 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 3a2c401596ee460784920a2c68477902 (table rebalancer_dynamic_table_new_name [id=74c89cc4ee6241f0b2066ff5cb2822b3]) successfully deleted
I20240501 14:00:42.135377 6350 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:42.135910 6350 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:42.136238 6350 ts_tablet_manager.cc:1367] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.252s user 0.042s sys 0.013s
I20240501 14:00:42.137132 6350 raft_consensus.cc:348] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:42.137488 6350 raft_consensus.cc:729] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:42.137755 6350 consensus_queue.cc:260] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:42.139266 6350 ts_tablet_manager.cc:1398] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:42.141026 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-b9d8d993ce834c1db4dd4418f7a85780 received from {username='slave'} at 127.28.17.7:46539
I20240501 14:00:42.141178 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-b9d8d993ce834c1db4dd4418f7a85780 on tablet b9d8d993ce834c1db4dd4418f7a85780 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:42.153056 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:42.164049 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41]), partition=
I20240501 14:00:42.164499 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.164507 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41]), partition=
I20240501 14:00:42.164844 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.167665 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41]), partition=
I20240501 14:00:42.168059 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.173710 6178 tablet_bootstrap.cc:492] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:42.173964 6346 tablet_bootstrap.cc:492] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:42.175465 6178 tablet_bootstrap.cc:654] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.175659 6346 tablet_bootstrap.cc:654] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.181370 5164 raft_consensus.cc:1203] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:42.183660 2535 raft_consensus.cc:1203] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:42.187582 2407 raft_consensus.cc:1203] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:42.190320 6178 tablet_bootstrap.cc:492] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:42.190634 6178 ts_tablet_manager.cc:1367] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20240501 14:00:42.191574 6178 raft_consensus.cc:348] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.191895 6178 raft_consensus.cc:374] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.192044 6178 raft_consensus.cc:729] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.192369 6178 consensus_queue.cc:260] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.193104 6178 ts_tablet_manager.cc:1398] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:42.220019 6057 tablet_bootstrap.cc:492] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:42.221807 6057 tablet_bootstrap.cc:654] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.226866 6057 tablet_bootstrap.cc:492] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:42.227051 6057 ts_tablet_manager.cc:1367] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 14:00:42.227759 6346 tablet_bootstrap.cc:492] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:42.227783 6057 raft_consensus.cc:348] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.227934 6346 ts_tablet_manager.cc:1367] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.054s user 0.020s sys 0.023s
I20240501 14:00:42.227962 6057 raft_consensus.cc:374] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.228025 6057 raft_consensus.cc:729] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.228276 6057 consensus_queue.cc:260] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.228811 6057 ts_tablet_manager.cc:1398] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:42.230036 6346 raft_consensus.cc:348] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.231374 6346 raft_consensus.cc:374] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.231555 6346 raft_consensus.cc:729] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.232088 6346 consensus_queue.cc:260] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.232897 6346 ts_tablet_manager.cc:1398] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:00:42.237304 4901 raft_consensus.cc:1203] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:42.260573 2147 consensus_queue.cc:237] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:42.263376 2535 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:42.263856 2080 consensus_peers.cc:485] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2efec972a06b48b99eb89d454fa214a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:42.263926 2407 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.264158 6062 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.264490 6374 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.268005 6062 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:42.273437 2535 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:42.275249 2001 catalog_manager.cc:5484] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:42.285086 2407 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:42.301023 6361 raft_consensus.cc:1050] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:00:42.301045 2275 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:00:42.301407 2535 consensus_queue.cc:237] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:42.301610 6377 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:42.302078 6361 consensus_queue.cc:237] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
W20240501 14:00:42.303966 2473 consensus_peers.cc:485] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8ae4d9e24a94b878627722ad94c0e35. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:42.304689 2407 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.305239 5727 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.305413 2147 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.307421 2147 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.307857 5727 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.308001 2535 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.309602 4901 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.309696 6472 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:42.310278 6361 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:42.310698 6354 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:42.314476 6453 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:42.316547 6362 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.317406 4901 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.318692 2147 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:42.324751 2001 catalog_manager.cc:5484] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.328512 2006 catalog_manager.cc:5484] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:42.337692 2535 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.341852 2407 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:42.349452 5033 consensus_queue.cc:237] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:00:42.349615 2147 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.350451 2407 raft_consensus.cc:1203] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:42.353809 2535 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.354904 6361 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.357093 4901 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.357940 6361 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.358543 6354 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.363348 2534 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.367583 4899 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:00:42.368006 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c685294d3fca4677bc1b1cf355dac3ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:42.368721 6377 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:42.369699 1999 catalog_manager.cc:5484] T c685294d3fca4677bc1b1cf355dac3ce P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:42.375898 6362 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.378423 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet c685294d3fca4677bc1b1cf355dac3ce with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:42.378834 6418 tablet_replica.cc:330] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:42.379052 6418 raft_consensus.cc:2229] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:42.379307 6418 raft_consensus.cc:2258] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:42.376624 4901 raft_consensus.cc:2932] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:42.383975 6418 ts_tablet_manager.cc:1875] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:42.386582 6062 consensus_queue.cc:1035] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.391654 6418 ts_tablet_manager.cc:1888] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:42.392835 2407 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:00:42.393743 6418 log.cc:1196] T c685294d3fca4677bc1b1cf355dac3ce P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/c685294d3fca4677bc1b1cf355dac3ce
I20240501 14:00:42.393667 6377 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:42.394856 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet c685294d3fca4677bc1b1cf355dac3ce (table rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]) successfully deleted
I20240501 14:00:42.417661 2147 consensus_queue.cc:237] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:42.420487 4901 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:00:42.422267 6374 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:42.423816 5033 raft_consensus.cc:1203] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:42.436353 2276 raft_consensus.cc:1261] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:00:42.437601 6062 consensus_queue.cc:1035] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:00:42.438324 6062 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:42.441510 6374 consensus_queue.cc:1035] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.446902 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:42.446995 6472 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.449745 2275 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:42.463521 2001 catalog_manager.cc:5484] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 5 to 7, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:42.464058 4900 raft_consensus.cc:2932] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:42.470026 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:42.470993 6493 tablet_replica.cc:330] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:42.471328 6493 raft_consensus.cc:2229] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:42.474957 6493 raft_consensus.cc:2258] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:42.476138 6493 ts_tablet_manager.cc:1875] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:42.484409 5965 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.488715 6493 ts_tablet_manager.cc:1888] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:00:42.488934 6493 log.cc:1196] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/44b7e8e96f4e4cc0bd2c824882ac4d84
I20240501 14:00:42.489821 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 44b7e8e96f4e4cc0bd2c824882ac4d84 (table rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]) successfully deleted
I20240501 14:00:42.493825 2534 consensus_queue.cc:237] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:42.496608 2406 raft_consensus.cc:1261] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.496634 2275 raft_consensus.cc:1261] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.497471 5727 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:42.497855 2472 consensus_peers.cc:485] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff4b3345715c44c091ed1073a8a706ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:42.498310 6453 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.501704 5976 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:42.502424 2406 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:42.512106 2003 catalog_manager.cc:5484] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:42.517642 6401 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.520392 2276 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:42.523290 6417 raft_consensus.cc:482] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:42.523478 6417 raft_consensus.cc:504] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.524174 6417 leader_election.cc:288] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:42.524852 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838ebc5d63e345d5b0fd2b2c2c66b58a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:42.525095 4900 raft_consensus.cc:2454] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:42.525941 5229 leader_election.cc:302] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:42.526818 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838ebc5d63e345d5b0fd2b2c2c66b58a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:42.527107 5164 raft_consensus.cc:2454] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:42.527963 6417 raft_consensus.cc:2781] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:42.528093 6417 raft_consensus.cc:482] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:42.528183 6417 raft_consensus.cc:3037] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.532261 6417 raft_consensus.cc:504] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.536078 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838ebc5d63e345d5b0fd2b2c2c66b58a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:42.536399 4900 raft_consensus.cc:3037] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.537691 2001 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:42.538744 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "838ebc5d63e345d5b0fd2b2c2c66b58a" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:42.538976 5164 raft_consensus.cc:3037] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.539129 6417 leader_election.cc:288] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:42.541779 5164 raft_consensus.cc:2454] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:42.542623 5228 leader_election.cc:302] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:42.544035 4900 raft_consensus.cc:2454] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:42.545197 6417 raft_consensus.cc:2781] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:42.545547 6417 raft_consensus.cc:686] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:42.545962 6417 consensus_queue.cc:237] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:42.549952 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6c6fcfebde46f18c55cf51c6b2f4f2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.550453 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6c6fcfebde46f18c55cf51c6b2f4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.555142 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 0689911a6a8147b983ca14edee551aa0 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.555616 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0689911a6a8147b983ca14edee551aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.555881 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6c6fcfebde46f18c55cf51c6b2f4f2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.556725 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6c6fcfebde46f18c55cf51c6b2f4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.560091 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4edc28f28641d4a3201cd896d97afe (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.560644 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4edc28f28641d4a3201cd896d97afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.563654 6507 tablet_bootstrap.cc:492] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:42.564185 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6c6fcfebde46f18c55cf51c6b2f4f2 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.566308 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 0689911a6a8147b983ca14edee551aa0 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.566324 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 0689911a6a8147b983ca14edee551aa0 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.566772 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0689911a6a8147b983ca14edee551aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.566785 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0689911a6a8147b983ca14edee551aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.567078 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6c6fcfebde46f18c55cf51c6b2f4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.567852 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4edc28f28641d4a3201cd896d97afe (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.568248 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4edc28f28641d4a3201cd896d97afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.568946 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 0d4edc28f28641d4a3201cd896d97afe (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:42.569342 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4edc28f28641d4a3201cd896d97afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.573673 6346 tablet_bootstrap.cc:492] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:42.575309 6346 tablet_bootstrap.cc:654] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.576421 6178 tablet_bootstrap.cc:492] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:42.578064 6178 tablet_bootstrap.cc:654] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.583259 6386 raft_consensus.cc:1050] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:42.584357 6386 consensus_queue.cc:237] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:42.591858 6057 tablet_bootstrap.cc:492] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:42.593583 6057 tablet_bootstrap.cc:654] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.599179 2003 catalog_manager.cc:5484] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:42.610168 6507 tablet_bootstrap.cc:654] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.614393 6509 tablet_bootstrap.cc:492] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:42.614643 5164 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:42.614820 2275 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:42.615408 5713 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:42.616003 5713 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:42.616439 6346 tablet_bootstrap.cc:492] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:42.616586 6346 ts_tablet_manager.cc:1367] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.011s
I20240501 14:00:42.616780 6509 tablet_bootstrap.cc:654] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.618868 6178 tablet_bootstrap.cc:492] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:42.619032 6510 tablet_bootstrap.cc:492] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:42.619045 6178 ts_tablet_manager.cc:1367] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.007s
I20240501 14:00:42.619880 6178 raft_consensus.cc:348] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.620097 6178 raft_consensus.cc:374] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.620177 6178 raft_consensus.cc:729] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.620446 6178 consensus_queue.cc:260] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.620782 6510 tablet_bootstrap.cc:654] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.620994 6178 ts_tablet_manager.cc:1398] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:42.621306 6178 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:42.623198 6057 tablet_bootstrap.cc:492] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:42.623358 6057 ts_tablet_manager.cc:1367] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.002s
I20240501 14:00:42.624122 6057 raft_consensus.cc:348] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.624293 2147 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:42.624359 6057 raft_consensus.cc:374] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.624441 6057 raft_consensus.cc:729] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.624722 6057 consensus_queue.cc:260] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.625017 4800 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:42.626252 6057 ts_tablet_manager.cc:1398] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:42.626571 6057 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:42.628620 6346 raft_consensus.cc:348] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.628854 6346 raft_consensus.cc:374] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.629014 6346 raft_consensus.cc:729] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.629297 6346 consensus_queue.cc:260] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.629868 6346 ts_tablet_manager.cc:1398] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 14:00:42.630102 6178 tablet_bootstrap.cc:654] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.630326 6057 tablet_bootstrap.cc:654] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.631896 6507 tablet_bootstrap.cc:492] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:42.632853 6507 ts_tablet_manager.cc:1367] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20240501 14:00:42.633675 6507 raft_consensus.cc:348] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.633885 6507 raft_consensus.cc:374] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.633971 6507 raft_consensus.cc:729] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.634238 6507 consensus_queue.cc:260] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.634908 6507 ts_tablet_manager.cc:1398] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:42.651907 6178 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:42.652119 6178 ts_tablet_manager.cc:1367] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240501 14:00:42.656765 6510 tablet_bootstrap.cc:492] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:42.657042 6510 ts_tablet_manager.cc:1367] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.015s
I20240501 14:00:42.657948 6510 raft_consensus.cc:348] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.658211 6178 raft_consensus.cc:348] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.658427 6178 raft_consensus.cc:374] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.658551 6178 raft_consensus.cc:729] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.658684 6510 raft_consensus.cc:374] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.658830 6510 raft_consensus.cc:729] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.659158 6510 consensus_queue.cc:260] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.659503 6178 consensus_queue.cc:260] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.662840 6178 ts_tablet_manager.cc:1398] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:00:42.664760 2534 consensus_queue.cc:237] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:42.665362 6510 ts_tablet_manager.cc:1398] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20240501 14:00:42.671372 2471 consensus_peers.cc:485] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1298bb5c8df3430b95d01ad23ff9cb71. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:42.671458 2276 raft_consensus.cc:1261] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.671802 2143 raft_consensus.cc:1261] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:42.672029 6453 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.672523 5976 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:42.676537 5976 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:42.680639 2276 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:42.689931 2003 catalog_manager.cc:5484] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:42.693893 6417 raft_consensus.cc:482] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:42.694119 6417 raft_consensus.cc:504] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.695314 6417 leader_election.cc:288] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:42.695694 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6c6fcfebde46f18c55cf51c6b2f4f2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:42.696206 2534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6c6fcfebde46f18c55cf51c6b2f4f2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:42.696460 2534 raft_consensus.cc:2454] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:42.697073 5228 leader_election.cc:302] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
W20240501 14:00:42.697681 5228 leader_election.cc:341] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:42.697743 5976 raft_consensus.cc:1050] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:42.698173 6417 raft_consensus.cc:2781] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:42.698348 6417 raft_consensus.cc:482] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:42.698536 6417 raft_consensus.cc:3037] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.698572 5976 consensus_queue.cc:237] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:42.705935 6417 raft_consensus.cc:504] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.707856 2534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6c6fcfebde46f18c55cf51c6b2f4f2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:42.708145 2534 raft_consensus.cc:3037] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.708202 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6c6fcfebde46f18c55cf51c6b2f4f2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
W20240501 14:00:42.709458 5228 leader_election.cc:341] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:42.709796 6417 leader_election.cc:288] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:42.711349 2534 raft_consensus.cc:2454] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:42.711863 5228 leader_election.cc:302] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters: a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:42.712136 6417 raft_consensus.cc:2781] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:42.712606 6417 raft_consensus.cc:686] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:42.712749 4900 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.713250 5713 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:42.714355 5164 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.714744 5727 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:42.715598 6057 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:42.715790 6057 ts_tablet_manager.cc:1367] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.005s
I20240501 14:00:42.716729 6057 raft_consensus.cc:348] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.716962 6057 raft_consensus.cc:374] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.716889 5727 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:42.717039 6057 raft_consensus.cc:729] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.717350 6057 consensus_queue.cc:260] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.718961 5163 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:42.722154 6057 ts_tablet_manager.cc:1398] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240501 14:00:42.725025 6530 ts_tablet_manager.cc:902] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:42.725551 1999 catalog_manager.cc:5484] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:42.728816 6417 consensus_queue.cc:237] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.730062 2147 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:42.732352 5296 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.734295 6509 tablet_bootstrap.cc:492] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:42.734280 5727 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:42.734542 6509 ts_tablet_manager.cc:1367] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.122s user 0.033s sys 0.044s
I20240501 14:00:42.735813 6509 raft_consensus.cc:348] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.736131 6509 raft_consensus.cc:374] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.736253 6509 raft_consensus.cc:729] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.736536 6509 consensus_queue.cc:260] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.737167 6509 ts_tablet_manager.cc:1398] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:42.739073 6509 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:42.741492 2275 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:42.741964 6509 tablet_bootstrap.cc:654] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:42.744993 2406 consensus_queue.cc:237] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:42.748425 5162 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:42.748493 6401 raft_consensus.cc:482] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:42.748643 6401 raft_consensus.cc:504] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.749330 6401 leader_election.cc:288] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:42.750016 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689911a6a8147b983ca14edee551aa0" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:42.750275 4898 raft_consensus.cc:2454] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:42.750809 6509 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:42.750880 5098 leader_election.cc:302] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:42.751030 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689911a6a8147b983ca14edee551aa0" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:42.751227 6401 raft_consensus.cc:2781] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:42.751272 2146 raft_consensus.cc:2454] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:42.751340 6401 raft_consensus.cc:482] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:42.751454 6401 raft_consensus.cc:3037] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.751751 6509 ts_tablet_manager.cc:1367] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20240501 14:00:42.752843 6509 raft_consensus.cc:348] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.754112 6509 raft_consensus.cc:374] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:42.755972 6509 raft_consensus.cc:729] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:42.756430 6509 consensus_queue.cc:260] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.755496 6401 raft_consensus.cc:504] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.757604 4800 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:42.757839 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689911a6a8147b983ca14edee551aa0" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:42.758049 4898 raft_consensus.cc:3037] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.759424 6509 ts_tablet_manager.cc:1398] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:42.760962 4898 raft_consensus.cc:2454] T 0689911a6a8147b983ca14edee551aa0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:42.761801 5098 leader_election.cc:302] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:42.762128 5965 raft_consensus.cc:2781] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:42.762377 5965 raft_consensus.cc:686] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:42.762943 5965 consensus_queue.cc:237] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:42.766008 6354 raft_consensus.cc:1050] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:00:42.766888 6354 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:42.785002 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0689911a6a8147b983ca14edee551aa0" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:42.785303 2147 raft_consensus.cc:3037] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.785198 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:42.788780 2147 raft_consensus.cc:2454] T 0689911a6a8147b983ca14edee551aa0 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:42.789898 6401 leader_election.cc:288] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:42.795349 6415 raft_consensus.cc:1050] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:00:42.796510 6415 consensus_queue.cc:237] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:00:42.804767 5713 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:42.809300 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9d8d993ce834c1db4dd4418f7a85780 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:42.811030 5162 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:42.815725 2003 catalog_manager.cc:5484] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:42.821136 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d8d993ce834c1db4dd4418f7a85780 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:42.821522 6418 tablet_replica.cc:330] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:42.821739 6418 raft_consensus.cc:2229] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:42.822535 6418 raft_consensus.cc:2258] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:42.825306 6418 ts_tablet_manager.cc:1875] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:42.835997 5162 raft_consensus.cc:1261] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:42.836819 6536 consensus_queue.cc:1035] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.840655 6418 ts_tablet_manager.cc:1888] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:42.840828 6418 log.cc:1196] T b9d8d993ce834c1db4dd4418f7a85780 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/b9d8d993ce834c1db4dd4418f7a85780
I20240501 14:00:42.841728 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet b9d8d993ce834c1db4dd4418f7a85780 (table rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]) successfully deleted
I20240501 14:00:42.842563 6543 ts_tablet_manager.cc:902] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:42.845940 2276 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:42.851378 6386 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:42.855527 2275 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:42.859812 2147 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:42.863492 6530 tablet_copy_client.cc:308] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:00:42.863492 6549 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:42.864104 6549 tablet.cc:1620] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:42.864372 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet 2efec972a06b48b99eb89d454fa214a3 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:46349)
I20240501 14:00:42.864535 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet 2efec972a06b48b99eb89d454fa214a3 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:46349: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-2efec972a06b48b99eb89d454fa214a3
I20240501 14:00:42.867529 2167 tablet_copy_source_session.cc:205] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:42.869660 6530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efec972a06b48b99eb89d454fa214a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:42.870414 6540 raft_consensus.cc:482] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:42.870757 6540 raft_consensus.cc:504] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.872135 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4edc28f28641d4a3201cd896d97afe" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:42.872377 2406 raft_consensus.cc:2454] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:42.872977 5228 leader_election.cc:302] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:42.873334 6536 raft_consensus.cc:2781] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:42.873484 6536 raft_consensus.cc:482] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:42.873642 6536 raft_consensus.cc:3037] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.874713 6550 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:42.876510 6550 tablet.cc:1620] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:42.876616 6536 raft_consensus.cc:504] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.878161 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4edc28f28641d4a3201cd896d97afe" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:42.878412 2406 raft_consensus.cc:3037] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.884485 2406 raft_consensus.cc:2454] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:42.885309 5228 leader_election.cc:302] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:42.886531 6530 tablet_copy_client.cc:783] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:42.887523 6530 tablet_copy_client.cc:647] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:42.889695 6530 tablet_copy_client.cc:523] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:42.894277 6536 leader_election.cc:288] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:42.895115 6536 raft_consensus.cc:2781] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:42.895447 6540 leader_election.cc:288] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:42.896131 6536 raft_consensus.cc:686] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:42.896747 6536 consensus_queue.cc:237] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:42.887585 6397 raft_consensus.cc:1050] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:00:42.899664 6530 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:42.899726 6397 consensus_queue.cc:237] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:42.907843 4898 raft_consensus.cc:1261] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:42.908231 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4edc28f28641d4a3201cd896d97afe" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:42.908488 4899 raft_consensus.cc:2454] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:42.908782 4901 raft_consensus.cc:1261] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.909191 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d4edc28f28641d4a3201cd896d97afe" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:42.909375 4898 raft_consensus.cc:3037] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:42.913683 1999 catalog_manager.cc:5484] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:42.916020 2003 catalog_manager.cc:5484] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:42.921504 5713 raft_consensus.cc:1050] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:42.922461 5713 consensus_queue.cc:237] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:42.924160 4898 raft_consensus.cc:2454] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:42.935163 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41] alter complete (version 1)
I20240501 14:00:42.939482 5033 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:42.939992 2276 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:42.940155 6554 ts_tablet_manager.cc:902] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:42.940511 2534 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:42.940577 4800 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:42.942124 6386 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:42.949419 6544 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:00:42.991905 2406 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:42.992800 6472 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:43.001827 6530 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:43.002686 6530 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:43.003041 6530 ts_tablet_manager.cc:1367] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.106s user 0.053s sys 0.000s
I20240501 14:00:43.003911 6530 raft_consensus.cc:348] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:43.004241 6530 raft_consensus.cc:729] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:43.004483 6530 consensus_queue.cc:260] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:43.005198 6530 ts_tablet_manager.cc:1398] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:43.006250 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-2efec972a06b48b99eb89d454fa214a3 received from {username='slave'} at 127.28.17.7:46349
I20240501 14:00:43.006412 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-2efec972a06b48b99eb89d454fa214a3 on tablet 2efec972a06b48b99eb89d454fa214a3 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:42.912225 6539 consensus_queue.cc:1035] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:42.919806 6417 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:42.966645 6543 tablet_copy_client.cc:308] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:43.015877 4899 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:43.016662 6472 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:43.030493 5164 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:43.031896 6354 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:43.047703 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:43.049326 6453 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:43.053073 4897 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:43.063148 5163 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:43.070191 1999 catalog_manager.cc:5484] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:43.089040 2535 consensus_queue.cc:237] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:43.093497 5161 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.096856 5976 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.099668 1999 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (table rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41]). Expected version 2 got 1
I20240501 14:00:43.104511 5976 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
W20240501 14:00:43.105715 6549 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.241s user 0.001s sys 0.000s
I20240501 14:00:43.113987 1999 catalog_manager.cc:5484] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:43.116447 5228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.17.3:33537, user_credentials={real_user=slave}} blocked reactor thread for 231143us
I20240501 14:00:43.117516 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet f8ae4d9e24a94b878627722ad94c0e35 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:41611)
I20240501 14:00:43.117681 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet f8ae4d9e24a94b878627722ad94c0e35 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:41611: session id = 5a4b9a96e9734baa8a03d6056bfa934c-f8ae4d9e24a94b878627722ad94c0e35
I20240501 14:00:43.120883 2558 tablet_copy_source_session.cc:205] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:43.123032 6543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ae4d9e24a94b878627722ad94c0e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.126252 5713 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:43.127424 6472 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.130175 6543 tablet_copy_client.cc:783] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:43.130429 6543 tablet_copy_client.cc:647] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:43.133025 2003 catalog_manager.cc:5484] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:43.136502 2535 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:43.141681 5033 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:43.141891 5161 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.150117 2006 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.151624 6545 raft_consensus.cc:482] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:43.151880 6545 raft_consensus.cc:504] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:43.152693 6545 leader_election.cc:288] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:43.157069 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6c6fcfebde46f18c55cf51c6b2f4f2" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:43.157395 2536 raft_consensus.cc:2379] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a64f34785a7b44fa8070b0452a4e0ddd in current term 1: Already voted for candidate 5a4b9a96e9734baa8a03d6056bfa934c in this term.
I20240501 14:00:43.158160 2408 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.163771 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6c6fcfebde46f18c55cf51c6b2f4f2" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:43.164853 2343 leader_election.cc:302] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:00:43.165509 5713 raft_consensus.cc:3037] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:43.168623 5713 raft_consensus.cc:2726] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:43.180512 5965 consensus_queue.cc:1035] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.189323 6549 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:43.189848 6549 tablet.cc:1620] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:43.190223 6550 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:43.190680 6550 tablet.cc:1620] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:43.193604 5161 raft_consensus.cc:1203] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:43.205199 2147 consensus_queue.cc:237] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:43.216820 6554 tablet_copy_client.cc:308] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:43.214809 2408 consensus_queue.cc:237] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:43.219341 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet ff4b3345715c44c091ed1073a8a706ff from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:50935)
I20240501 14:00:43.219620 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet ff4b3345715c44c091ed1073a8a706ff from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:50935: session id = f8c60f2bbac541dcba4424ab726f9ad9-ff4b3345715c44c091ed1073a8a706ff
I20240501 14:00:43.224646 1999 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (table rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41]). Expected version 2 got 1
I20240501 14:00:43.216856 5965 consensus_queue.cc:1035] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:43.225682 2082 consensus_peers.cc:485] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387698a371cc42eab970f2f4471bbf5b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:43.226874 2407 raft_consensus.cc:1261] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.227573 6489 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.231736 2536 raft_consensus.cc:1261] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.232448 6489 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.242811 6543 tablet_copy_client.cc:523] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:43.245321 6551 consensus_queue.cc:1035] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.246435 6543 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:43.246733 2538 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:43.248265 5713 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:43.251269 6489 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:43.253060 2407 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:43.267102 5164 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:43.275208 5162 raft_consensus.cc:1261] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:43.275779 2534 raft_consensus.cc:1261] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:43.276978 4900 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.277891 6551 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:43.276139 6530 ts_tablet_manager.cc:902] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:43.285995 6530 tablet_copy_client.cc:308] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:43.286890 2557 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 1298bb5c8df3430b95d01ad23ff9cb71 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:35785)
I20240501 14:00:43.287081 2557 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 1298bb5c8df3430b95d01ad23ff9cb71 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:35785: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-1298bb5c8df3430b95d01ad23ff9cb71
I20240501 14:00:43.288584 6551 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:43.288225 2533 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:43.290644 2557 tablet_copy_source_session.cc:205] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:43.290359 6551 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:43.292263 6530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1298bb5c8df3430b95d01ad23ff9cb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.259979 2001 catalog_manager.cc:5484] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:43.284379 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb1fa60c64034b7d8c7b4879823b0d83 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:43.295941 4897 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:43.298771 5032 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:43.294602 2533 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:43.300843 6530 tablet_copy_client.cc:783] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:43.301077 6530 tablet_copy_client.cc:647] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:43.303901 6530 tablet_copy_client.cc:523] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:43.305205 4900 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.305718 6453 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.307288 4898 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:43.309978 6551 consensus_queue.cc:1035] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.311316 6530 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:43.314011 6386 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:43.316955 6545 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:43.321002 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c74bacbf4464bab9c403e2153953474 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:43.323945 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41] alter complete (version 2)
I20240501 14:00:43.323675 5032 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:43.331740 2005 catalog_manager.cc:5484] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:43.340605 2003 catalog_manager.cc:5484] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:43.356726 2000 catalog_manager.cc:5484] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:43.361601 2558 tablet_copy_source_session.cc:205] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:43.377097 6543 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:43.380596 6543 tablet_bootstrap.cc:492] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:00:43.381116 6543 ts_tablet_manager.cc:1367] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.135s user 0.047s sys 0.004s
I20240501 14:00:43.382014 2538 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:43.382493 6543 raft_consensus.cc:348] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:43.382762 6543 raft_consensus.cc:729] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:00:43.382961 6543 consensus_queue.cc:260] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:43.384634 5032 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:43.385123 6543 ts_tablet_manager.cc:1398] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:43.386417 2557 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-f8ae4d9e24a94b878627722ad94c0e35 received from {username='slave'} at 127.28.17.8:41611
I20240501 14:00:43.386595 2557 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-f8ae4d9e24a94b878627722ad94c0e35 on tablet f8ae4d9e24a94b878627722ad94c0e35 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:43.389240 6576 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.389839 4900 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.394614 6551 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.395345 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf051cecef424db5accfce66623346b5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:43.395937 2408 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.396728 6472 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.398773 2408 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.411739 6540 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:43.412322 2001 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:43.419358 6554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4b3345715c44c091ed1073a8a706ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.424602 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1fa60c64034b7d8c7b4879823b0d83 with delete_type TABLET_DATA_TOMBSTONED (TS 5a4b9a96e9734baa8a03d6056bfa934c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:43.425331 6600 tablet_replica.cc:330] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:43.425580 6600 raft_consensus.cc:2229] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.425796 6600 pending_rounds.cc:62] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Trying to abort 1 pending ops.
I20240501 14:00:43.425899 6600 pending_rounds.cc:68] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022887086890950656 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "eb1fa60c64034b7d8c7b4879823b0d83" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } } }
I20240501 14:00:43.426388 6600 raft_consensus.cc:2894] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:00:43.426543 6600 raft_consensus.cc:2866] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:00:43.426666 6600 raft_consensus.cc:2258] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.429010 6554 tablet_copy_client.cc:783] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:43.429356 6554 tablet_copy_client.cc:647] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:43.430737 6600 ts_tablet_manager.cc:1875] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:43.434921 6530 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:43.435382 6530 tablet_bootstrap.cc:492] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:43.435638 6530 ts_tablet_manager.cc:1367] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.124s user 0.049s sys 0.003s
I20240501 14:00:43.444830 6530 raft_consensus.cc:348] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:43.445111 6530 raft_consensus.cc:729] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:43.445355 6530 consensus_queue.cc:260] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:43.448875 6600 ts_tablet_manager.cc:1888] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:43.449430 6600 log.cc:1196] T eb1fa60c64034b7d8c7b4879823b0d83 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/eb1fa60c64034b7d8c7b4879823b0d83
I20240501 14:00:43.451191 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet eb1fa60c64034b7d8c7b4879823b0d83 (table rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]) successfully deleted
I20240501 14:00:43.452571 6530 ts_tablet_manager.cc:1398] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240501 14:00:43.456995 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-1298bb5c8df3430b95d01ad23ff9cb71 received from {username='slave'} at 127.28.17.7:35785
I20240501 14:00:43.457242 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-1298bb5c8df3430b95d01ad23ff9cb71 on tablet 1298bb5c8df3430b95d01ad23ff9cb71 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:43.467283 6554 tablet_copy_client.cc:523] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:43.471446 6554 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:43.480830 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:43.502362 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet bf051cecef424db5accfce66623346b5 with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
W20240501 14:00:43.524005 6397 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.070s sys 0.133s
W20240501 14:00:43.524173 6397 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.070s sys 0.133s
I20240501 14:00:43.525907 6596 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:43.529531 6596 tablet.cc:1620] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:43.535414 5161 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
W20240501 14:00:43.537894 2275 consensus_meta.cc:287] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent flushing consensus metadata: real 0.535s user 0.001s sys 0.001s
I20240501 14:00:43.550081 6605 tablet_replica.cc:330] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:43.550968 6605 raft_consensus.cc:2229] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.551611 6605 raft_consensus.cc:2258] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.556536 6605 ts_tablet_manager.cc:1875] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:43.557089 1999 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (table rebalancer_dynamic_table [id=63c202953e2a4f908aa5d7909652eb41]). Expected version 2 got 1
W20240501 14:00:43.561298 2276 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.156s sys 0.215s
W20240501 14:00:43.561470 2276 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.156s sys 0.215s
I20240501 14:00:43.590478 6605 ts_tablet_manager.cc:1888] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:43.591017 6605 log.cc:1196] T bf051cecef424db5accfce66623346b5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/bf051cecef424db5accfce66623346b5
I20240501 14:00:43.591950 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet bf051cecef424db5accfce66623346b5 (table rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]) successfully deleted
I20240501 14:00:43.595887 6596 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:43.596376 6596 tablet.cc:1620] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:43.603950 6549 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:43.608180 6554 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:43.608659 6554 tablet_bootstrap.cc:492] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:43.608964 6554 ts_tablet_manager.cc:1367] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.138s user 0.045s sys 0.001s
I20240501 14:00:43.609750 6554 raft_consensus.cc:348] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:43.610002 6554 raft_consensus.cc:729] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:43.610220 6554 consensus_queue.cc:260] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:43.610441 6550 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:43.610826 6554 ts_tablet_manager.cc:1398] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:43.611572 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-ff4b3345715c44c091ed1073a8a706ff received from {username='slave'} at 127.28.17.5:50935
I20240501 14:00:43.611701 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-ff4b3345715c44c091ed1073a8a706ff on tablet ff4b3345715c44c091ed1073a8a706ff with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:43.613834 6596 tablet.cc:1722] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:43.629540 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c74bacbf4464bab9c403e2153953474 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:43.633347 6614 tablet_replica.cc:330] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:43.633846 6614 raft_consensus.cc:2229] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.634536 6614 pending_rounds.cc:62] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Trying to abort 1 pending ops.
I20240501 14:00:43.634792 6614 pending_rounds.cc:68] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022887087808167936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6c74bacbf4464bab9c403e2153953474" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } } }
I20240501 14:00:43.635922 6614 raft_consensus.cc:2894] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:00:43.636564 6614 raft_consensus.cc:2866] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:00:43.637171 6614 raft_consensus.cc:2258] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.642632 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=63c202953e2a4f908aa5d7909652eb41] alter complete (version 3)
I20240501 14:00:43.642773 6614 ts_tablet_manager.cc:1875] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:43.645092 5296 consensus_queue.cc:237] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:00:43.648578 5164 raft_consensus.cc:1261] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.649479 2533 raft_consensus.cc:1261] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.650585 6540 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.651183 6540 consensus_queue.cc:1035] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.655820 6417 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:43.662196 2533 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:43.666405 5164 raft_consensus.cc:2932] T f9b8c92907e843409e37919a1a0e96af P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:43.673056 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9b8c92907e843409e37919a1a0e96af with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:43.676307 2005 catalog_manager.cc:5484] T f9b8c92907e843409e37919a1a0e96af P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:00:43.681039 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b8c92907e843409e37919a1a0e96af with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:43.690665 6618 tablet_replica.cc:330] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:43.690989 6618 raft_consensus.cc:2229] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.691433 6618 raft_consensus.cc:2258] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.693966 2406 raft_consensus.cc:1261] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:43.695181 6618 ts_tablet_manager.cc:1875] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:43.698762 6611 consensus_queue.cc:1035] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:43.709836 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:43.710570 6618 ts_tablet_manager.cc:1888] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:43.715814 6618 log.cc:1196] T f9b8c92907e843409e37919a1a0e96af P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/f9b8c92907e843409e37919a1a0e96af
I20240501 14:00:43.718963 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet f9b8c92907e843409e37919a1a0e96af (table rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]) successfully deleted
I20240501 14:00:43.721619 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 566acdef1d114b8aac9b55ccfa03da35 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.721873 5012 tablet_service.cc:1467] Processing CreateTablet for tablet eb9790fa13fe4cb286f0fb7f730da336 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.722360 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9790fa13fe4cb286f0fb7f730da336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.723011 2255 tablet_service.cc:1467] Processing CreateTablet for tablet efbd148f37bb46f0a4bf0fc562fff597 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.723032 2516 tablet_service.cc:1467] Processing CreateTablet for tablet efbd148f37bb46f0a4bf0fc562fff597 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.723290 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 566acdef1d114b8aac9b55ccfa03da35 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.723416 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbd148f37bb46f0a4bf0fc562fff597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.723445 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbd148f37bb46f0a4bf0fc562fff597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.723657 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566acdef1d114b8aac9b55ccfa03da35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.728191 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566acdef1d114b8aac9b55ccfa03da35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.731021 5142 tablet_service.cc:1467] Processing CreateTablet for tablet efbd148f37bb46f0a4bf0fc562fff597 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.731503 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbd148f37bb46f0a4bf0fc562fff597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.735244 5143 tablet_service.cc:1467] Processing CreateTablet for tablet eb9790fa13fe4cb286f0fb7f730da336 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.735677 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9790fa13fe4cb286f0fb7f730da336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.735987 4879 tablet_service.cc:1467] Processing CreateTablet for tablet eb9790fa13fe4cb286f0fb7f730da336 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.736429 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9790fa13fe4cb286f0fb7f730da336. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.748442 6614 ts_tablet_manager.cc:1888] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:43.748606 6614 log.cc:1196] T 6c74bacbf4464bab9c403e2153953474 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c74bacbf4464bab9c403e2153953474
I20240501 14:00:43.750873 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 6c74bacbf4464bab9c403e2153953474 (table rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]) successfully deleted
I20240501 14:00:43.751403 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 566acdef1d114b8aac9b55ccfa03da35 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:43.751848 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566acdef1d114b8aac9b55ccfa03da35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.753832 6623 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:43.756222 6623 tablet_bootstrap.cc:654] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.721967 6617 ts_tablet_manager.cc:902] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:43.763633 5164 raft_consensus.cc:1203] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:43.767284 2406 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.770399 2408 consensus_queue.cc:237] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:43.770684 6489 raft_consensus.cc:1050] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:43.771500 6489 consensus_queue.cc:237] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:43.775204 2001 catalog_manager.cc:5484] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.780298 2001 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:43.780642 2001 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:43.786038 2001 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 838ebc5d63e345d5b0fd2b2c2c66b58a
I20240501 14:00:43.787456 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:43.787559 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:43.787655 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 838ebc5d63e345d5b0fd2b2c2c66b58a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:43 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:43.787806 6618 tablet_replica.cc:330] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:43.787854 6605 tablet_replica.cc:330] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:43.788033 6618 raft_consensus.cc:2229] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.788033 6605 raft_consensus.cc:2229] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.788270 6618 raft_consensus.cc:2258] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.788283 6605 raft_consensus.cc:2258] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.788586 6600 tablet_replica.cc:330] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:43.788779 6600 raft_consensus.cc:2229] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:43.789170 6600 raft_consensus.cc:2258] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.789721 6618 ts_tablet_manager.cc:1875] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.790330 6605 ts_tablet_manager.cc:1875] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.797302 6600 ts_tablet_manager.cc:1875] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:43.798692 6618 ts_tablet_manager.cc:1888] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:43.798962 6618 log.cc:1196] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/838ebc5d63e345d5b0fd2b2c2c66b58a
I20240501 14:00:43.799436 6618 ts_tablet_manager.cc:1909] T 838ebc5d63e345d5b0fd2b2c2c66b58a P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:43.801362 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (table rebalancer_dynamic_table_new_name [id=63c202953e2a4f908aa5d7909652eb41]) successfully deleted
W20240501 14:00:43.809317 2343 consensus_peers.cc:485] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6cca7b2b71045bf98e6eacd14e007d6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:43.765889 6609 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.811491 2147 raft_consensus.cc:1261] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.814673 4899 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.815640 2537 raft_consensus.cc:1261] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.818164 6545 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.819450 6544 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.822458 2275 consensus_queue.cc:237] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:43.830299 2406 raft_consensus.cc:1261] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.831166 6397 consensus_queue.cc:1035] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.835188 2406 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.837028 6605 ts_tablet_manager.cc:1888] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:43.842250 4899 raft_consensus.cc:1261] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:43.843075 6601 consensus_queue.cc:1035] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:43.845173 4901 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.850243 6609 raft_consensus.cc:2932] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.854323 5295 raft_consensus.cc:1203] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:43.839932 6605 log.cc:1196] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/838ebc5d63e345d5b0fd2b2c2c66b58a
I20240501 14:00:43.856004 6605 ts_tablet_manager.cc:1909] T 838ebc5d63e345d5b0fd2b2c2c66b58a P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:43.857298 2533 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:43.857733 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (table rebalancer_dynamic_table_new_name [id=63c202953e2a4f908aa5d7909652eb41]) successfully deleted
I20240501 14:00:43.858018 6369 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:43.858284 6600 ts_tablet_manager.cc:1888] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:43.858561 6600 log.cc:1196] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/838ebc5d63e345d5b0fd2b2c2c66b58a
I20240501 14:00:43.859273 6600 ts_tablet_manager.cc:1909] T 838ebc5d63e345d5b0fd2b2c2c66b58a P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:43.861866 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 838ebc5d63e345d5b0fd2b2c2c66b58a (table rebalancer_dynamic_table_new_name [id=63c202953e2a4f908aa5d7909652eb41]) successfully deleted
I20240501 14:00:43.863075 5164 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:43.863932 6369 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:43.871572 2001 catalog_manager.cc:5484] T 9935f0f87637467091ff85989503af28 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:43.877364 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 9935f0f87637467091ff85989503af28 with delete_type TABLET_DATA_TOMBSTONED (TS 5a4b9a96e9734baa8a03d6056bfa934c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:43.877652 6386 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:43.885195 2406 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:43.886581 6369 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:43.886849 2001 catalog_manager.cc:5484] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:43.892388 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:43.896871 6600 tablet_replica.cc:330] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:43.898375 6600 raft_consensus.cc:2229] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:43.898842 6600 pending_rounds.cc:62] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Trying to abort 1 pending ops.
I20240501 14:00:43.887426 2533 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:43.899065 6600 pending_rounds.cc:68] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022887087714869248 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9935f0f87637467091ff85989503af28" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } } }
I20240501 14:00:43.899936 6600 raft_consensus.cc:2894] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:00:43.900188 6600 raft_consensus.cc:2866] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:00:43.900418 6600 raft_consensus.cc:2258] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:43.905721 2147 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:43.910180 6624 tablet_bootstrap.cc:492] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:43.911899 6621 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:43.918025 6621 tablet_bootstrap.cc:654] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:43.920892 6629 log.cc:924] Time spent T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20240501 14:00:43.922672 6626 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:43.924511 6626 tablet_bootstrap.cc:654] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.930251 6622 tablet_bootstrap.cc:492] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:43.932184 6622 tablet_bootstrap.cc:654] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.937366 4881 tablet_service.cc:1467] Processing CreateTablet for tablet ee444373d372455d9edd5e6c0009ba37 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c]), partition=
I20240501 14:00:43.937762 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee444373d372455d9edd5e6c0009ba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.939555 5012 tablet_service.cc:1467] Processing CreateTablet for tablet ee444373d372455d9edd5e6c0009ba37 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c]), partition=
I20240501 14:00:43.939894 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee444373d372455d9edd5e6c0009ba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.941859 6600 ts_tablet_manager.cc:1875] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:43.942451 5143 tablet_service.cc:1467] Processing CreateTablet for tablet ee444373d372455d9edd5e6c0009ba37 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c]), partition=
I20240501 14:00:43.942891 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee444373d372455d9edd5e6c0009ba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:43.945823 6624 tablet_bootstrap.cc:654] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.949771 6625 tablet_bootstrap.cc:492] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:43.955777 6623 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:43.956037 6623 ts_tablet_manager.cc:1367] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.202s user 0.004s sys 0.000s
I20240501 14:00:43.957027 6623 raft_consensus.cc:348] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:43.962955 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9935f0f87637467091ff85989503af28 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:43.971174 6369 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:43.977713 5161 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:43.979065 2538 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:43.982846 6623 raft_consensus.cc:374] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.982971 6623 raft_consensus.cc:729] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.983312 6623 consensus_queue.cc:260] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:43.983968 4901 raft_consensus.cc:1203] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:43.985692 1999 catalog_manager.cc:5484] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:43.992033 6622 tablet_bootstrap.cc:492] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:43.992223 6622 ts_tablet_manager.cc:1367] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20240501 14:00:43.993222 6622 raft_consensus.cc:348] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:43.993454 6622 raft_consensus.cc:374] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:43.993541 6622 raft_consensus.cc:729] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:43.993781 6622 consensus_queue.cc:260] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:43.994372 6622 ts_tablet_manager.cc:1398] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:43.994670 6622 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:43.996309 6622 tablet_bootstrap.cc:654] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:43.997377 6617 tablet_copy_client.cc:308] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:00:43.998349 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet 387698a371cc42eab970f2f4471bbf5b from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:50703)
I20240501 14:00:43.998569 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet 387698a371cc42eab970f2f4471bbf5b from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:50703: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-387698a371cc42eab970f2f4471bbf5b
I20240501 14:00:44.002216 2167 tablet_copy_source_session.cc:205] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:44.003916 6623 ts_tablet_manager.cc:1398] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20240501 14:00:44.004032 6617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387698a371cc42eab970f2f4471bbf5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.004354 6623 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:44.006213 6623 tablet_bootstrap.cc:654] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.012470 6623 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:44.012693 6623 ts_tablet_manager.cc:1367] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240501 14:00:44.013669 6623 raft_consensus.cc:348] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.013935 6623 raft_consensus.cc:374] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.014001 6623 raft_consensus.cc:729] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.014261 6623 consensus_queue.cc:260] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.014883 6623 ts_tablet_manager.cc:1398] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:44.018859 6617 tablet_copy_client.cc:783] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:44.019178 6617 tablet_copy_client.cc:647] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:44.020725 6621 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:44.020903 6621 ts_tablet_manager.cc:1367] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.109s user 0.005s sys 0.000s
I20240501 14:00:44.023520 6621 raft_consensus.cc:348] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:44.023790 6621 raft_consensus.cc:374] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.023882 6621 raft_consensus.cc:729] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.024121 6621 consensus_queue.cc:260] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:44.029002 6626 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:44.028663 2406 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.031566 6626 ts_tablet_manager.cc:1367] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.109s user 0.000s sys 0.004s
I20240501 14:00:44.034473 6624 tablet_bootstrap.cc:492] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:44.034847 6624 ts_tablet_manager.cc:1367] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.001s
I20240501 14:00:44.036113 6626 raft_consensus.cc:348] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.036446 6625 tablet_bootstrap.cc:654] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.039924 6600 ts_tablet_manager.cc:1888] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:44.040102 6621 ts_tablet_manager.cc:1398] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:00:44.040580 6626 raft_consensus.cc:374] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.037346 6624 raft_consensus.cc:348] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.040926 6624 raft_consensus.cc:374] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.041196 6626 raft_consensus.cc:729] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.041393 6624 raft_consensus.cc:729] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.042016 6626 consensus_queue.cc:260] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.042428 6624 consensus_queue.cc:260] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.044548 6624 ts_tablet_manager.cc:1398] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:00:44.045109 6626 ts_tablet_manager.cc:1398] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:00:44.046662 6600 log.cc:1196] T 9935f0f87637467091ff85989503af28 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/9935f0f87637467091ff85989503af28
I20240501 14:00:44.048584 6624 tablet_bootstrap.cc:492] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:44.049647 6625 tablet_bootstrap.cc:492] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:44.049891 6625 ts_tablet_manager.cc:1367] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.100s user 0.000s sys 0.005s
I20240501 14:00:44.050693 6622 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:44.050912 6622 ts_tablet_manager.cc:1367] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.056s user 0.018s sys 0.027s
I20240501 14:00:44.051271 6625 raft_consensus.cc:348] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.051578 6625 raft_consensus.cc:374] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.051714 6625 raft_consensus.cc:729] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.051734 6622 raft_consensus.cc:348] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.051941 6622 raft_consensus.cc:374] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.052016 6622 raft_consensus.cc:729] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.052253 6622 consensus_queue.cc:260] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.052239 6625 consensus_queue.cc:260] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.052796 6622 ts_tablet_manager.cc:1398] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:44.052902 6625 ts_tablet_manager.cc:1398] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:44.053114 6622 tablet_bootstrap.cc:492] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:44.055136 6622 tablet_bootstrap.cc:654] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.059495 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 9935f0f87637467091ff85989503af28 (table rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]) successfully deleted
I20240501 14:00:44.062067 2538 consensus_queue.cc:237] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:44.062626 6624 tablet_bootstrap.cc:654] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.067037 6617 tablet_copy_client.cc:523] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:44.067502 6625 tablet_bootstrap.cc:492] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:44.069751 6625 tablet_bootstrap.cc:654] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.070711 6617 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:44.070749 2147 consensus_queue.cc:237] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:44.073336 5161 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:44.074476 6062 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.080325 6062 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.084184 5161 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.091451 6622 tablet_bootstrap.cc:492] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:44.091730 6622 ts_tablet_manager.cc:1367] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.012s
I20240501 14:00:44.092649 6622 raft_consensus.cc:348] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.092978 6622 raft_consensus.cc:374] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.093120 6622 raft_consensus.cc:729] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.093448 6622 consensus_queue.cc:260] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.094143 6622 ts_tablet_manager.cc:1398] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:44.100790 2000 catalog_manager.cc:5484] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.107426 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 2efec972a06b48b99eb89d454fa214a3 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:44.111166 6609 raft_consensus.cc:482] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.111336 6609 raft_consensus.cc:504] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.112155 6609 leader_election.cc:288] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:44.112515 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566acdef1d114b8aac9b55ccfa03da35" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:44.112728 2143 raft_consensus.cc:2454] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:44.114982 6661 tablet_replica.cc:330] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:44.115325 6661 raft_consensus.cc:2229] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.115674 2211 leader_election.cc:302] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:44.116223 6602 raft_consensus.cc:2781] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:44.116320 6602 raft_consensus.cc:482] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:44.116423 6602 raft_consensus.cc:3037] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.116948 6661 raft_consensus.cc:2258] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.120703 6661 ts_tablet_manager.cc:1875] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:44.127401 6661 ts_tablet_manager.cc:1888] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:44.126284 6602 raft_consensus.cc:504] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.137660 6602 leader_election.cc:288] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:44.138644 6661 log.cc:1196] T 2efec972a06b48b99eb89d454fa214a3 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/2efec972a06b48b99eb89d454fa214a3
I20240501 14:00:44.138649 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566acdef1d114b8aac9b55ccfa03da35" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:44.138903 2143 raft_consensus.cc:3037] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.139834 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 2efec972a06b48b99eb89d454fa214a3 (table rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]) successfully deleted
I20240501 14:00:44.141865 2143 raft_consensus.cc:2454] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:44.142750 2211 leader_election.cc:302] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:44.145226 6617 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:44.145735 6617 tablet_bootstrap.cc:492] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:44.146099 6617 ts_tablet_manager.cc:1367] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.075s user 0.047s sys 0.008s
I20240501 14:00:44.147118 6617 raft_consensus.cc:348] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:44.147516 6617 raft_consensus.cc:729] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:44.147792 6617 consensus_queue.cc:260] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:44.148504 6617 ts_tablet_manager.cc:1398] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:44.149636 6602 raft_consensus.cc:2781] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:44.150823 6602 raft_consensus.cc:686] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:44.151146 6602 consensus_queue.cc:237] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.153172 6654 raft_consensus.cc:482] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.153339 6654 raft_consensus.cc:504] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.155174 6654 leader_election.cc:288] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:44.155894 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566acdef1d114b8aac9b55ccfa03da35" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:44.156162 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-387698a371cc42eab970f2f4471bbf5b received from {username='slave'} at 127.28.17.6:50703
I20240501 14:00:44.156280 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-387698a371cc42eab970f2f4471bbf5b on tablet 387698a371cc42eab970f2f4471bbf5b with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:44.156841 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566acdef1d114b8aac9b55ccfa03da35" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:44.158668 2143 raft_consensus.cc:2379] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in current term 1: Already voted for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in this term.
I20240501 14:00:44.159309 4967 leader_election.cc:302] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887
I20240501 14:00:44.159808 6654 raft_consensus.cc:3037] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.163314 6654 raft_consensus.cc:2726] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 14:00:44.179822 2080 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.7:41601, user_credentials={real_user=slave}} blocked reactor thread for 81498.1us
I20240501 14:00:44.181231 2275 raft_consensus.cc:1261] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:44.181635 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2efec972a06b48b99eb89d454fa214a3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:44.182981 2408 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:00:44.183521 2472 consensus_peers.cc:485] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 304b465c0a99456790bfc8f262b61ecd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:44.184440 6453 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:44.187670 1999 catalog_manager.cc:5484] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:44.189606 6489 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.191776 2406 raft_consensus.cc:1261] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:44.193514 2408 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.194576 6607 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:44.197849 6654 raft_consensus.cc:482] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.198042 6654 raft_consensus.cc:504] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.199962 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9790fa13fe4cb286f0fb7f730da336" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:44.200219 4901 raft_consensus.cc:2454] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:44.200721 4968 leader_election.cc:302] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:44.201048 6653 raft_consensus.cc:2781] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:44.201149 6653 raft_consensus.cc:482] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:44.201243 6653 raft_consensus.cc:3037] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.202190 6654 leader_election.cc:288] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:44.204847 6653 raft_consensus.cc:504] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.204950 6453 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:44.206636 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9790fa13fe4cb286f0fb7f730da336" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:44.206853 4901 raft_consensus.cc:3037] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.208716 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9790fa13fe4cb286f0fb7f730da336" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:44.208925 5164 raft_consensus.cc:3037] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.209544 4901 raft_consensus.cc:2454] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:44.210119 4968 leader_election.cc:302] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:44.211434 6654 raft_consensus.cc:2781] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:44.211863 5164 raft_consensus.cc:2454] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:44.211901 6654 raft_consensus.cc:686] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:44.212203 6654 consensus_queue.cc:237] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.213411 6653 leader_election.cc:288] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:44.213841 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9790fa13fe4cb286f0fb7f730da336" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:44.214082 5161 raft_consensus.cc:2362] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1. Re-sending same reply.
I20240501 14:00:44.217608 6625 tablet_bootstrap.cc:492] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:44.217821 6625 ts_tablet_manager.cc:1367] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.150s user 0.003s sys 0.000s
I20240501 14:00:44.218008 2005 catalog_manager.cc:5484] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:44.218842 6625 raft_consensus.cc:348] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.219916 6625 raft_consensus.cc:374] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.220057 6625 raft_consensus.cc:729] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.220656 6625 consensus_queue.cc:260] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.221387 6625 ts_tablet_manager.cc:1398] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:44.223346 6624 tablet_bootstrap.cc:492] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:44.223572 6624 ts_tablet_manager.cc:1367] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.175s user 0.038s sys 0.064s
I20240501 14:00:44.223594 2275 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:44.224380 6624 raft_consensus.cc:348] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.224591 6624 raft_consensus.cc:374] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.224653 6624 raft_consensus.cc:729] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.224880 6624 consensus_queue.cc:260] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.225450 6624 ts_tablet_manager.cc:1398] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:44.225761 6624 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:44.227382 6624 tablet_bootstrap.cc:654] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.229426 6624 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:44.229029 2406 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:44.229763 6624 ts_tablet_manager.cc:1367] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:44.230592 6624 raft_consensus.cc:348] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:44.230823 6624 raft_consensus.cc:374] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.230927 6624 raft_consensus.cc:729] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.231266 6624 consensus_queue.cc:260] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:44.233243 6624 ts_tablet_manager.cc:1398] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:44.235411 1999 catalog_manager.cc:5484] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:44.247140 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566acdef1d114b8aac9b55ccfa03da35" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:44.247274 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566acdef1d114b8aac9b55ccfa03da35" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:44.247449 5032 raft_consensus.cc:2454] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:44.256605 2406 consensus_queue.cc:237] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:44.258790 2275 raft_consensus.cc:1261] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:44.258872 2538 raft_consensus.cc:1261] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:44.259016 2341 consensus_peers.cc:485] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5f7a0845388479b9d5f5f12d776c978. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:44.259356 5713 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:44.259688 6544 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:44.262626 6602 raft_consensus.cc:482] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.262781 6602 raft_consensus.cc:504] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:44.262645 5713 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:44.263571 6602 leader_election.cc:288] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:44.263945 2538 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:44.268033 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:44.268303 2533 raft_consensus.cc:2454] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:44.268321 1997 catalog_manager.cc:5484] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:44.268868 2211 leader_election.cc:302] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:44.269173 6602 raft_consensus.cc:2781] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:44.269304 6602 raft_consensus.cc:482] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:44.269011 2275 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:44.269407 6602 raft_consensus.cc:3037] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.272061 6602 raft_consensus.cc:504] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:44.275583 6602 leader_election.cc:288] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:44.277464 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:44.277952 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:44.278203 5161 raft_consensus.cc:2454] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:44.278530 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:44.278690 5164 raft_consensus.cc:3037] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.279981 2538 raft_consensus.cc:3037] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.281332 5164 raft_consensus.cc:2454] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:44.282066 2211 leader_election.cc:302] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:44.282482 6602 raft_consensus.cc:2781] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:44.282692 6602 raft_consensus.cc:686] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:44.283066 6602 consensus_queue.cc:237] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:44.284041 5965 raft_consensus.cc:482] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.284235 5965 raft_consensus.cc:504] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.284925 5965 leader_election.cc:288] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:44.285656 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee444373d372455d9edd5e6c0009ba37" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:44.285929 5033 raft_consensus.cc:2454] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:44.286128 2538 raft_consensus.cc:2454] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:44.286438 5099 leader_election.cc:302] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:44.287556 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee444373d372455d9edd5e6c0009ba37" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:44.287822 4901 raft_consensus.cc:2454] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:44.288658 5965 raft_consensus.cc:2781] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:44.288759 5965 raft_consensus.cc:482] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:44.288868 5965 raft_consensus.cc:3037] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.291750 5965 raft_consensus.cc:504] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.292066 1999 catalog_manager.cc:5484] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:44.292855 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee444373d372455d9edd5e6c0009ba37" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:44.293085 5033 raft_consensus.cc:3037] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.296918 5033 raft_consensus.cc:2454] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:44.297544 5099 leader_election.cc:302] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:44.297904 6501 raft_consensus.cc:2781] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:44.298394 6501 raft_consensus.cc:686] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:44.298801 6501 consensus_queue.cc:237] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.299309 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee444373d372455d9edd5e6c0009ba37" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:44.299526 4901 raft_consensus.cc:3037] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:44.304081 4901 raft_consensus.cc:2454] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:44.304822 5965 leader_election.cc:288] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:44.307572 6687 ts_tablet_manager.cc:902] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:44.307889 1999 catalog_manager.cc:5484] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:44.319190 6687 tablet_copy_client.cc:308] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:44.320394 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet b6cca7b2b71045bf98e6eacd14e007d6 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:53689)
I20240501 14:00:44.320581 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet b6cca7b2b71045bf98e6eacd14e007d6 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:53689: session id = 5a4b9a96e9734baa8a03d6056bfa934c-b6cca7b2b71045bf98e6eacd14e007d6
I20240501 14:00:44.322988 2428 tablet_copy_source_session.cc:205] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:44.324350 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cca7b2b71045bf98e6eacd14e007d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.330651 6687 tablet_copy_client.cc:783] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:44.330893 6687 tablet_copy_client.cc:647] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:44.332679 6687 tablet_copy_client.cc:523] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:44.335693 6687 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:44.366545 6453 raft_consensus.cc:1050] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:44.367533 6453 consensus_queue.cc:237] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:44.375187 5164 raft_consensus.cc:1261] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.377647 5976 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.381990 2275 raft_consensus.cc:1261] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.382005 2143 raft_consensus.cc:1261] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.383172 5976 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.384467 5976 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.398751 6687 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:44.398777 5976 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.399317 6687 tablet_bootstrap.cc:492] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:00:44.399614 6687 ts_tablet_manager.cc:1367] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.064s user 0.045s sys 0.005s
I20240501 14:00:44.400372 6687 raft_consensus.cc:348] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:44.400662 6687 raft_consensus.cc:729] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:00:44.400856 6687 consensus_queue.cc:260] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:44.403317 6687 ts_tablet_manager.cc:1398] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:44.403831 2274 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.407601 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-b6cca7b2b71045bf98e6eacd14e007d6 received from {username='slave'} at 127.28.17.8:53689
I20240501 14:00:44.407794 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-b6cca7b2b71045bf98e6eacd14e007d6 on tablet b6cca7b2b71045bf98e6eacd14e007d6 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:44.409698 2143 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.410933 1997 catalog_manager.cc:5484] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.411044 5164 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.416725 2538 consensus_queue.cc:237] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:44.420406 2143 raft_consensus.cc:1261] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:44.420951 6453 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.421605 5161 raft_consensus.cc:1261] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:44.422590 6715 consensus_queue.cc:1035] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.423529 5976 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.427516 5164 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.430943 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1298bb5c8df3430b95d01ad23ff9cb71 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:44.432394 2143 raft_consensus.cc:2932] T 1298bb5c8df3430b95d01ad23ff9cb71 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:44.434360 6607 raft_consensus.cc:1050] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:00:44.435261 6607 consensus_queue.cc:237] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:44.438319 2406 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.439327 5293 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.439539 6715 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.439992 6714 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.440972 2143 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.441589 6714 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.448885 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:44.434551 2006 catalog_manager.cc:5484] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:44.457501 4899 raft_consensus.cc:1261] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:44.458190 6501 consensus_queue.cc:1035] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.459988 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 1298bb5c8df3430b95d01ad23ff9cb71 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:44.467749 5033 raft_consensus.cc:1261] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:44.475842 6501 consensus_queue.cc:1035] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.485525 6731 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:44.487059 6731 tablet.cc:1620] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:44.488430 1997 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet ee444373d372455d9edd5e6c0009ba37 (table rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c]). Expected version 1 got 0
I20240501 14:00:44.499909 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c] alter complete (version 1)
I20240501 14:00:44.504559 6726 tablet_replica.cc:330] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:44.509493 1997 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet ee444373d372455d9edd5e6c0009ba37 (table rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c]). Expected version 1 got 0
I20240501 14:00:44.522805 6726 raft_consensus.cc:2229] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.523268 6726 raft_consensus.cc:2258] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.524587 6726 ts_tablet_manager.cc:1875] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:44.505775 6730 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:44.527443 6730 tablet.cc:1620] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:44.533407 6715 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.534229 5295 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.539517 2143 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.540113 1997 catalog_manager.cc:5484] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.544070 2006 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet ee444373d372455d9edd5e6c0009ba37 (table rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c]). Expected version 1 got 0
I20240501 14:00:44.546734 6733 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:44.547829 6733 tablet.cc:1620] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:44.562541 6726 ts_tablet_manager.cc:1888] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:44.565326 2406 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.568732 2538 consensus_queue.cc:237] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:44.571259 5293 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:44.572012 6714 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.574528 6726 log.cc:1196] T 1298bb5c8df3430b95d01ad23ff9cb71 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/1298bb5c8df3430b95d01ad23ff9cb71
I20240501 14:00:44.574990 2406 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:44.574707 6607 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.575975 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 1298bb5c8df3430b95d01ad23ff9cb71 (table rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]) successfully deleted
I20240501 14:00:44.576814 5976 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.578604 5293 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.578724 2406 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.585536 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8ae4d9e24a94b878627722ad94c0e35 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:44.586966 2006 catalog_manager.cc:5484] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:44.591089 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ae4d9e24a94b878627722ad94c0e35 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:44.594219 6654 consensus_queue.cc:1035] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.599071 6741 tablet_replica.cc:330] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:44.599328 6741 raft_consensus.cc:2229] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.599575 6741 raft_consensus.cc:2258] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.600314 6715 raft_consensus.cc:1050] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:44.601080 6715 consensus_queue.cc:237] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:44.604301 6472 consensus_queue.cc:1035] T eb9790fa13fe4cb286f0fb7f730da336 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.605012 6741 ts_tablet_manager.cc:1875] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:44.605863 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:44.609987 4901 raft_consensus.cc:1261] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.610402 2274 raft_consensus.cc:1261] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.610697 6715 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.614347 2406 raft_consensus.cc:1261] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:44.617100 6741 ts_tablet_manager.cc:1888] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:44.617272 6741 log.cc:1196] T f8ae4d9e24a94b878627722ad94c0e35 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8ae4d9e24a94b878627722ad94c0e35
I20240501 14:00:44.617501 6730 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:44.618052 6730 tablet.cc:1620] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:44.618255 6607 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.618674 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet f8ae4d9e24a94b878627722ad94c0e35 (table rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]) successfully deleted
I20240501 14:00:44.619269 6607 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:44.619966 6733 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:44.620460 6733 tablet.cc:1620] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:44.626626 6731 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:44.627161 6731 tablet.cc:1620] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:44.639053 6714 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.642113 2406 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.644353 4901 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.644708 2274 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.645813 2006 catalog_manager.cc:5484] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:44.640902 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=40089bbab5be406b9e627cbc9a5bcf0c] alter complete (version 2)
I20240501 14:00:44.650866 2538 consensus_queue.cc:237] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:44.652949 4901 raft_consensus.cc:1261] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:44.653743 6715 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.658463 6602 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.660486 2275 raft_consensus.cc:1261] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:44.660496 5976 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.662922 6453 consensus_queue.cc:1035] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:44.664552 2275 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.666075 4900 raft_consensus.cc:2932] T ff4b3345715c44c091ed1073a8a706ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.674870 6752 ts_tablet_manager.cc:902] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:44.677664 5293 raft_consensus.cc:1203] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:44.681166 1999 catalog_manager.cc:5484] T ff4b3345715c44c091ed1073a8a706ff P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:44.685220 5032 raft_consensus.cc:1203] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:44.685532 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff4b3345715c44c091ed1073a8a706ff with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:44.705058 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4b3345715c44c091ed1073a8a706ff with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:44.716029 6609 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.738334 6602 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.745425 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:44.757150 6730 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:44.758641 6733 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:44.759150 6731 tablet.cc:1722] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:44.768656 6602 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:44.768932 2006 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=40089bbab5be406b9e627cbc9a5bcf0c] alter complete (version 3)
I20240501 14:00:44.775089 6761 ts_tablet_manager.cc:902] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:44.799023 6756 tablet_replica.cc:330] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:44.799355 6756 raft_consensus.cc:2229] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.799710 6756 raft_consensus.cc:2258] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.806123 6756 ts_tablet_manager.cc:1875] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:44.809736 6761 tablet_copy_client.cc:308] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:44.810683 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 304b465c0a99456790bfc8f262b61ecd from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:50935)
I20240501 14:00:44.810892 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 304b465c0a99456790bfc8f262b61ecd from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:50935: session id = f8c60f2bbac541dcba4424ab726f9ad9-304b465c0a99456790bfc8f262b61ecd
I20240501 14:00:44.813468 2558 tablet_copy_source_session.cc:205] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:44.814411 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:44.817627 6756 ts_tablet_manager.cc:1888] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:44.817770 6756 log.cc:1196] T ff4b3345715c44c091ed1073a8a706ff P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/ff4b3345715c44c091ed1073a8a706ff
I20240501 14:00:44.818724 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet ff4b3345715c44c091ed1073a8a706ff (table rebalance_test_table_15 [id=47c9a2251e7740f88134b83425f3c01a]) successfully deleted
I20240501 14:00:44.820369 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304b465c0a99456790bfc8f262b61ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.821775 6752 tablet_copy_client.cc:308] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:44.822788 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet e5f7a0845388479b9d5f5f12d776c978 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:46539)
I20240501 14:00:44.822961 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet e5f7a0845388479b9d5f5f12d776c978 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:46539: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-e5f7a0845388479b9d5f5f12d776c978
I20240501 14:00:44.824944 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 9a74335efeb04521baf1720a14093643 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.825459 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a74335efeb04521baf1720a14093643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.825599 2428 tablet_copy_source_session.cc:205] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:44.826443 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 913ae5fda9e842e1b3a6d7dc418a5bfd (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.826915 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913ae5fda9e842e1b3a6d7dc418a5bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.827487 6752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f7a0845388479b9d5f5f12d776c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.827502 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed332b3f7304bf69bc348d230ece128 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.827517 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 9a74335efeb04521baf1720a14093643 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.827916 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed332b3f7304bf69bc348d230ece128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.827910 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a74335efeb04521baf1720a14093643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.830966 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 913ae5fda9e842e1b3a6d7dc418a5bfd (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.831398 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913ae5fda9e842e1b3a6d7dc418a5bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.832655 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed332b3f7304bf69bc348d230ece128 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.833086 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed332b3f7304bf69bc348d230ece128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.835944 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 913ae5fda9e842e1b3a6d7dc418a5bfd (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.836429 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913ae5fda9e842e1b3a6d7dc418a5bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.839618 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 9a74335efeb04521baf1720a14093643 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.840137 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a74335efeb04521baf1720a14093643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.844935 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed332b3f7304bf69bc348d230ece128 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:44.845444 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed332b3f7304bf69bc348d230ece128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:44.847267 6761 tablet_copy_client.cc:783] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:44.847509 6761 tablet_copy_client.cc:647] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:44.849282 6761 tablet_copy_client.cc:523] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:44.853091 6761 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:44.861207 6752 tablet_copy_client.cc:783] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:44.862090 6752 tablet_copy_client.cc:647] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:44.869594 6752 tablet_copy_client.cc:523] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:44.877924 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:44.878162 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:44.880621 6752 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:44.884166 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet ee444373d372455d9edd5e6c0009ba37
I20240501 14:00:44.885852 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet ee444373d372455d9edd5e6c0009ba37 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:44.885488 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet ee444373d372455d9edd5e6c0009ba37 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:44.886693 5014 tablet_service.cc:1514] Processing DeleteTablet for tablet ee444373d372455d9edd5e6c0009ba37 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:44 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:44.902911 6780 tablet_replica.cc:330] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:44.903303 6780 raft_consensus.cc:2229] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.903592 6780 raft_consensus.cc:2258] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.904670 6780 ts_tablet_manager.cc:1875] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.905982 6779 tablet_replica.cc:330] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:44.906561 6779 raft_consensus.cc:2229] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:44.913847 6779 raft_consensus.cc:2258] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.915591 6779 ts_tablet_manager.cc:1875] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.918715 6780 ts_tablet_manager.cc:1888] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:44.919520 6780 log.cc:1196] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/ee444373d372455d9edd5e6c0009ba37
I20240501 14:00:44.920276 6780 ts_tablet_manager.cc:1909] T ee444373d372455d9edd5e6c0009ba37 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:44.921423 6778 tablet_replica.cc:330] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:44.921762 6778 raft_consensus.cc:2229] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:44.921905 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet ee444373d372455d9edd5e6c0009ba37 (table rebalancer_dynamic_table_new_name [id=40089bbab5be406b9e627cbc9a5bcf0c]) successfully deleted
I20240501 14:00:44.922245 6778 raft_consensus.cc:2258] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:44.922609 6785 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:44.924880 6778 ts_tablet_manager.cc:1875] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:44.926359 6785 tablet_bootstrap.cc:654] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.952060 6778 ts_tablet_manager.cc:1888] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:44.955513 6783 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:44.958462 6783 tablet_bootstrap.cc:654] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.964623 6785 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:44.965746 6785 ts_tablet_manager.cc:1367] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.000s
I20240501 14:00:44.966846 6785 raft_consensus.cc:348] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:44.966532 6783 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:44.968231 6785 raft_consensus.cc:374] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.968282 6779 ts_tablet_manager.cc:1888] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:44.968343 6783 ts_tablet_manager.cc:1367] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20240501 14:00:44.968441 6779 log.cc:1196] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/ee444373d372455d9edd5e6c0009ba37
I20240501 14:00:44.968624 6785 raft_consensus.cc:729] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.968924 6779 ts_tablet_manager.cc:1909] T ee444373d372455d9edd5e6c0009ba37 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:44.969452 6785 consensus_queue.cc:260] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:44.970118 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet ee444373d372455d9edd5e6c0009ba37 (table rebalancer_dynamic_table_new_name [id=40089bbab5be406b9e627cbc9a5bcf0c]) successfully deleted
I20240501 14:00:44.969203 6783 raft_consensus.cc:348] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.971715 6785 ts_tablet_manager.cc:1398] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:44.971938 6782 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:44.972565 6785 tablet_bootstrap.cc:492] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:44.972842 6778 log.cc:1196] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/ee444373d372455d9edd5e6c0009ba37
I20240501 14:00:44.971263 6783 raft_consensus.cc:374] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.974539 6778 ts_tablet_manager.cc:1909] T ee444373d372455d9edd5e6c0009ba37 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:44.974565 6783 raft_consensus.cc:729] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.974936 6783 consensus_queue.cc:260] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:44.975620 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet ee444373d372455d9edd5e6c0009ba37 (table rebalancer_dynamic_table_new_name [id=40089bbab5be406b9e627cbc9a5bcf0c]) successfully deleted
I20240501 14:00:44.978013 6782 tablet_bootstrap.cc:654] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.978490 6785 tablet_bootstrap.cc:654] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.979770 6775 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:44.980279 6776 tablet_bootstrap.cc:492] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:44.981560 6785 tablet_bootstrap.cc:492] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:44.981828 6785 ts_tablet_manager.cc:1367] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:00:44.982728 6783 ts_tablet_manager.cc:1398] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 14:00:44.984627 6785 raft_consensus.cc:348] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.984979 6785 raft_consensus.cc:374] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:44.985134 6785 raft_consensus.cc:729] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:44.985460 6785 consensus_queue.cc:260] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:44.987200 6785 ts_tablet_manager.cc:1398] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:44.988760 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:44.993880 6776 tablet_bootstrap.cc:654] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.998179 6775 tablet_bootstrap.cc:654] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:44.999575 6609 raft_consensus.cc:482] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:44.999737 6609 raft_consensus.cc:504] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.000455 6609 leader_election.cc:288] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:45.001263 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed332b3f7304bf69bc348d230ece128" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
W20240501 14:00:45.002074 2211 leader_election.cc:341] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:45.003110 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 948b150d59024a58b813b9353d665394 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]), partition=
I20240501 14:00:45.003530 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948b150d59024a58b813b9353d665394. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.003855 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 948b150d59024a58b813b9353d665394 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]), partition=
I20240501 14:00:45.004205 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948b150d59024a58b813b9353d665394. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.009884 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed332b3f7304bf69bc348d230ece128" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:00:45.010668 2212 leader_election.cc:341] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:45.010833 2212 leader_election.cc:302] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:45.011586 6764 raft_consensus.cc:2726] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:45.013180 6782 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:45.013350 6782 ts_tablet_manager.cc:1367] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.002s
I20240501 14:00:45.014801 6782 raft_consensus.cc:348] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.015578 6782 raft_consensus.cc:374] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.015614 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 948b150d59024a58b813b9353d665394 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]), partition=
I20240501 14:00:45.015724 6782 raft_consensus.cc:729] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.016050 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948b150d59024a58b813b9353d665394. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.015966 6782 consensus_queue.cc:260] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.016563 6782 ts_tablet_manager.cc:1398] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:45.017915 6782 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:45.019624 6782 tablet_bootstrap.cc:654] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.023759 6801 tablet_bootstrap.cc:492] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:45.027556 6782 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:45.027616 6801 tablet_bootstrap.cc:654] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.027858 6782 ts_tablet_manager.cc:1367] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 14:00:45.029546 6782 raft_consensus.cc:348] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.029912 6782 raft_consensus.cc:374] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.030227 6782 raft_consensus.cc:729] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.030606 6782 consensus_queue.cc:260] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.040881 6801 tablet_bootstrap.cc:492] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:45.041707 6801 ts_tablet_manager.cc:1367] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240501 14:00:45.043071 6801 raft_consensus.cc:348] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.044473 6775 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:45.044647 6761 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:45.044694 6775 ts_tablet_manager.cc:1367] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.003s
I20240501 14:00:45.044839 6801 raft_consensus.cc:374] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.045132 6801 raft_consensus.cc:729] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.045598 6775 raft_consensus.cc:348] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.045821 6775 raft_consensus.cc:374] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.045912 6775 raft_consensus.cc:729] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.045723 6801 consensus_queue.cc:260] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.046159 6775 consensus_queue.cc:260] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.047314 6775 ts_tablet_manager.cc:1398] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:45.053727 6801 ts_tablet_manager.cc:1398] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:00:45.056134 6776 tablet_bootstrap.cc:492] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:45.056411 6776 ts_tablet_manager.cc:1367] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.003s
I20240501 14:00:45.057273 6761 tablet_bootstrap.cc:492] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:45.057276 6776 raft_consensus.cc:348] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.057590 6776 raft_consensus.cc:374] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.057729 6776 raft_consensus.cc:729] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.058048 6776 consensus_queue.cc:260] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.058168 6761 ts_tablet_manager.cc:1367] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.205s user 0.052s sys 0.004s
I20240501 14:00:45.059540 6761 raft_consensus.cc:348] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:45.059962 6776 ts_tablet_manager.cc:1398] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:45.060314 6776 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:45.062914 6775 tablet_bootstrap.cc:492] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:45.066088 6776 tablet_bootstrap.cc:654] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.053359 6782 ts_tablet_manager.cc:1398] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.025s user 0.002s sys 0.002s
I20240501 14:00:45.068056 6782 tablet_bootstrap.cc:492] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:45.069746 6782 tablet_bootstrap.cc:654] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.071559 6761 raft_consensus.cc:729] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:45.072144 6761 consensus_queue.cc:260] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:45.073961 6761 ts_tablet_manager.cc:1398] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 14:00:45.075886 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-304b465c0a99456790bfc8f262b61ecd received from {username='slave'} at 127.28.17.5:50935
I20240501 14:00:45.075992 6775 tablet_bootstrap.cc:654] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.076031 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-304b465c0a99456790bfc8f262b61ecd on tablet 304b465c0a99456790bfc8f262b61ecd with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:45.078032 6417 raft_consensus.cc:482] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.078208 6417 raft_consensus.cc:504] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.078982 6417 leader_election.cc:288] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:45.079641 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b150d59024a58b813b9353d665394" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:45.082036 6782 tablet_bootstrap.cc:492] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:45.082186 6782 ts_tablet_manager.cc:1367] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240501 14:00:45.083001 6782 raft_consensus.cc:348] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.083199 6782 raft_consensus.cc:374] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.083285 6782 raft_consensus.cc:729] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.083271 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b150d59024a58b813b9353d665394" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:45.083526 6782 consensus_queue.cc:260] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
W20240501 14:00:45.087198 5229 leader_election.cc:341] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:45.088174 6782 ts_tablet_manager.cc:1398] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20240501 14:00:45.088459 5228 leader_election.cc:341] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:45.088624 5228 leader_election.cc:302] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:45.094938 6775 tablet_bootstrap.cc:492] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:45.094939 6776 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:45.095203 6775 ts_tablet_manager.cc:1367] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.003s
I20240501 14:00:45.095269 6776 ts_tablet_manager.cc:1367] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.002s
I20240501 14:00:45.095748 6417 raft_consensus.cc:2726] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:45.095799 6545 raft_consensus.cc:1050] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:00:45.096005 6775 raft_consensus.cc:348] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.096344 6775 raft_consensus.cc:374] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.096467 6775 raft_consensus.cc:729] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.096962 6775 consensus_queue.cc:260] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.097183 6545 consensus_queue.cc:237] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:45.096252 6776 raft_consensus.cc:348] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.098119 6776 raft_consensus.cc:374] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.098205 6776 raft_consensus.cc:729] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.098670 6775 ts_tablet_manager.cc:1398] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:45.099031 6775 tablet_bootstrap.cc:492] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:45.100379 6776 consensus_queue.cc:260] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.103199 6775 tablet_bootstrap.cc:654] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.105739 6776 ts_tablet_manager.cc:1398] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:00:45.111920 6775 tablet_bootstrap.cc:492] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:45.112104 6775 ts_tablet_manager.cc:1367] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20240501 14:00:45.113000 6775 raft_consensus.cc:348] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.113245 6775 raft_consensus.cc:374] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.113386 6775 raft_consensus.cc:729] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.114166 6775 consensus_queue.cc:260] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.116367 6775 ts_tablet_manager.cc:1398] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:00:45.117149 6653 raft_consensus.cc:482] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.117327 6653 raft_consensus.cc:504] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.117588 2147 raft_consensus.cc:1261] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.118022 6653 leader_election.cc:288] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:45.118441 5293 raft_consensus.cc:1261] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.119236 6545 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.119601 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913ae5fda9e842e1b3a6d7dc418a5bfd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:45.119745 2538 raft_consensus.cc:1261] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.119817 5164 raft_consensus.cc:2454] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:45.120486 4967 leader_election.cc:302] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:45.120585 6545 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.121654 6545 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.121731 6653 raft_consensus.cc:2781] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.121688 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913ae5fda9e842e1b3a6d7dc418a5bfd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:45.121829 6653 raft_consensus.cc:482] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.121879 2147 raft_consensus.cc:2454] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:45.121927 6653 raft_consensus.cc:3037] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.124071 6653 raft_consensus.cc:504] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.124621 6653 leader_election.cc:288] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:45.125864 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913ae5fda9e842e1b3a6d7dc418a5bfd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:45.126081 5164 raft_consensus.cc:3037] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.128257 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913ae5fda9e842e1b3a6d7dc418a5bfd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:45.128499 2147 raft_consensus.cc:3037] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.129705 5164 raft_consensus.cc:2454] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:45.130247 4967 leader_election.cc:302] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:45.130633 6653 raft_consensus.cc:2781] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.131217 6653 raft_consensus.cc:686] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:45.131596 6653 consensus_queue.cc:237] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.138190 2147 raft_consensus.cc:2454] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:45.143075 6686 raft_consensus.cc:482] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.143296 6686 raft_consensus.cc:504] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.144111 6686 leader_election.cc:288] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:45.144687 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b150d59024a58b813b9353d665394" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:45.144940 5293 raft_consensus.cc:2454] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:45.144901 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b150d59024a58b813b9353d665394" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:45.145154 4901 raft_consensus.cc:2454] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:45.145602 5099 leader_election.cc:302] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:45.145884 6686 raft_consensus.cc:2781] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.145998 6686 raft_consensus.cc:482] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.146095 6686 raft_consensus.cc:3037] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.149266 6686 raft_consensus.cc:504] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.152880 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b150d59024a58b813b9353d665394" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:45.153177 4901 raft_consensus.cc:3037] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.158924 4901 raft_consensus.cc:2454] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:45.160089 5098 leader_election.cc:302] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:45.160470 5965 raft_consensus.cc:2781] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.164125 6686 leader_election.cc:288] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:45.164721 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948b150d59024a58b813b9353d665394" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:45.164953 5293 raft_consensus.cc:3037] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.149423 6752 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:45.167670 6752 tablet_bootstrap.cc:492] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:45.167898 5293 raft_consensus.cc:2454] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:45.173480 6752 ts_tablet_manager.cc:1367] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.293s user 0.055s sys 0.000s
I20240501 14:00:45.174587 6752 raft_consensus.cc:348] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:45.175690 6752 raft_consensus.cc:729] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:45.176007 6752 consensus_queue.cc:260] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:45.176553 6545 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:45.177058 2538 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:45.177573 6752 ts_tablet_manager.cc:1398] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:45.178375 5965 raft_consensus.cc:686] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:45.178684 5965 consensus_queue.cc:237] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.180630 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-e5f7a0845388479b9d5f5f12d776c978 received from {username='slave'} at 127.28.17.7:46539
I20240501 14:00:45.180779 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-e5f7a0845388479b9d5f5f12d776c978 on tablet e5f7a0845388479b9d5f5f12d776c978 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:45.182791 2147 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:45.184655 5293 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:45.189024 1999 catalog_manager.cc:5484] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:45.197407 2406 consensus_queue.cc:237] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:45.201053 5164 raft_consensus.cc:1203] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:45.206168 6660 raft_consensus.cc:1050] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:45.207199 6660 consensus_queue.cc:237] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:45.211920 5295 raft_consensus.cc:1261] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.212884 5713 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.215826 2538 raft_consensus.cc:1261] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.215669 5713 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:45.218642 6545 consensus_queue.cc:1035] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.220759 2003 catalog_manager.cc:5484] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.220789 2538 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:45.222330 5295 raft_consensus.cc:2932] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:45.224002 1999 catalog_manager.cc:5484] T b6cca7b2b71045bf98e6eacd14e007d6 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:45.224880 2533 raft_consensus.cc:1261] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.225694 5033 raft_consensus.cc:1261] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.226398 6660 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.226979 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6cca7b2b71045bf98e6eacd14e007d6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:45.228390 1997 catalog_manager.cc:5484] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:45.230083 6489 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.231431 2406 raft_consensus.cc:1261] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.233634 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet b6cca7b2b71045bf98e6eacd14e007d6 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:45.234632 6827 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240501 14:00:45.239720 6829 tablet_replica.cc:330] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:45.240020 6829 raft_consensus.cc:2229] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.240374 6829 raft_consensus.cc:2258] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.241498 6829 ts_tablet_manager.cc:1875] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:45.243891 4901 raft_consensus.cc:1203] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:45.251348 6829 ts_tablet_manager.cc:1888] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:45.251636 6829 log.cc:1196] T b6cca7b2b71045bf98e6eacd14e007d6 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/b6cca7b2b71045bf98e6eacd14e007d6
I20240501 14:00:45.252648 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet b6cca7b2b71045bf98e6eacd14e007d6 (table rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]) successfully deleted
I20240501 14:00:45.255436 6593 raft_consensus.cc:482] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.255607 6593 raft_consensus.cc:504] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.255293 6827 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:45.256392 6593 leader_election.cc:288] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:45.257920 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74335efeb04521baf1720a14093643" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:45.258189 2143 raft_consensus.cc:2454] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:45.259393 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74335efeb04521baf1720a14093643" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:45.259689 5032 raft_consensus.cc:2454] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:45.262048 4834 leader_election.cc:302] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:45.262164 2538 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:45.263085 6593 raft_consensus.cc:2781] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.263196 6593 raft_consensus.cc:482] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.263269 6593 raft_consensus.cc:3037] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.263018 5033 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:45.266989 6593 raft_consensus.cc:504] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.268893 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74335efeb04521baf1720a14093643" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:45.269194 5032 raft_consensus.cc:3037] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.271952 5032 raft_consensus.cc:2454] T 9a74335efeb04521baf1720a14093643 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:45.272055 2406 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:45.272747 4836 leader_election.cc:302] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:45.273102 6800 raft_consensus.cc:2781] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.273247 6800 raft_consensus.cc:686] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:45.273536 6800 consensus_queue.cc:237] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.278044 2001 catalog_manager.cc:5484] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:45.286931 6593 leader_election.cc:288] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:45.287564 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74335efeb04521baf1720a14093643" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:45.287848 2143 raft_consensus.cc:3037] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.288800 6821 raft_consensus.cc:482] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:45.288975 6821 raft_consensus.cc:504] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.289677 6821 leader_election.cc:288] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:45.290557 2143 raft_consensus.cc:2454] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:45.290957 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed332b3f7304bf69bc348d230ece128" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:45.291208 2275 raft_consensus.cc:2454] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:45.291142 2147 consensus_queue.cc:237] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:45.291678 5097 leader_election.cc:302] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:45.294776 6821 raft_consensus.cc:2781] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:45.294919 6821 raft_consensus.cc:482] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:45.295006 6821 raft_consensus.cc:3037] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.295009 2406 raft_consensus.cc:1261] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.295156 2001 catalog_manager.cc:5484] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.296777 6489 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.297850 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed332b3f7304bf69bc348d230ece128" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:45.298111 4901 raft_consensus.cc:2454] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:45.298460 6821 raft_consensus.cc:504] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.299700 5032 raft_consensus.cc:1261] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.299568 6827 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:45.300175 2408 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:45.303803 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 387698a371cc42eab970f2f4471bbf5b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:45.303910 6660 consensus_queue.cc:1035] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.304867 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed332b3f7304bf69bc348d230ece128" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:45.305105 2275 raft_consensus.cc:3037] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.306797 5032 raft_consensus.cc:2932] T 387698a371cc42eab970f2f4471bbf5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:45.308331 2275 raft_consensus.cc:2454] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:45.308822 6821 leader_election.cc:288] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:45.308879 5097 leader_election.cc:302] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:45.309281 6821 raft_consensus.cc:2781] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.310016 6821 raft_consensus.cc:686] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:45.310282 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed332b3f7304bf69bc348d230ece128" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:45.310628 4901 raft_consensus.cc:3037] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.310894 6821 consensus_queue.cc:237] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.310820 1997 catalog_manager.cc:5484] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.313823 4901 raft_consensus.cc:2454] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:45.315676 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 387698a371cc42eab970f2f4471bbf5b with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:45.316558 6836 tablet_replica.cc:330] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:45.316843 6836 raft_consensus.cc:2229] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.317221 6836 raft_consensus.cc:2258] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.318102 6836 ts_tablet_manager.cc:1875] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:45.323194 1997 catalog_manager.cc:5484] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.327459 6836 ts_tablet_manager.cc:1888] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:45.327713 6836 log.cc:1196] T 387698a371cc42eab970f2f4471bbf5b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/387698a371cc42eab970f2f4471bbf5b
I20240501 14:00:45.329408 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 387698a371cc42eab970f2f4471bbf5b (table rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]) successfully deleted
I20240501 14:00:45.363202 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:45.376408 5295 raft_consensus.cc:1261] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:45.376754 4901 raft_consensus.cc:1261] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:45.377038 5965 consensus_queue.cc:1035] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.377382 6821 consensus_queue.cc:1035] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.383569 1997 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]). Expected version 1 got 0
I20240501 14:00:45.386442 6858 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:45.387311 6858 tablet.cc:1620] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:45.389250 1997 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]). Expected version 1 got 0
I20240501 14:00:45.396816 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0] alter complete (version 1)
I20240501 14:00:45.399827 6857 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:45.401830 6857 tablet.cc:1620] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:45.402139 2001 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]). Expected version 1 got 0
I20240501 14:00:45.404541 2001 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]). Expected version 1 got 0
I20240501 14:00:45.408749 6862 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:45.410612 6862 tablet.cc:1620] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:45.411293 2001 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0]). Expected version 1 got 0
I20240501 14:00:45.453310 2275 consensus_queue.cc:237] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:45.455941 2538 raft_consensus.cc:1261] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.456007 2408 raft_consensus.cc:1261] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:45.456255 2211 consensus_peers.cc:485] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73940622ad734b88af31e9fa0c1000e5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.456570 6764 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.456890 6878 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.459684 6764 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:45.463850 2538 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:45.465073 2408 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:45.469409 2001 catalog_manager.cc:5484] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.497562 2408 consensus_queue.cc:237] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:45.499783 2001 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:45.500041 2538 raft_consensus.cc:1261] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.500108 2275 raft_consensus.cc:1261] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:45.500312 2343 consensus_peers.cc:485] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcb6951344ad49b3aee27205e898c347. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.500618 6545 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.501116 6545 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.504933 5713 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:45.509096 6857 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:45.509465 6858 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:45.509670 6857 tablet.cc:1620] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:45.509953 6858 tablet.cc:1620] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:45.515543 2538 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:45.519351 6862 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:45.519924 6862 tablet.cc:1620] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:45.522678 2275 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:00:45.523612 2004 catalog_manager.cc:5484] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.525879 2001 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=0651835ef96943f69dbd61ba74cc3ab0] alter complete (version 2)
I20240501 14:00:45.545219 5164 consensus_queue.cc:237] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:45.547716 2147 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:45.547916 5295 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:00:45.548143 5099 consensus_peers.cc:485] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44a4e92b144f47b5800506f4bc3eadb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.548354 5965 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:45.548682 6686 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:45.551054 5965 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:45.551692 5295 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:45.554462 2147 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:45.559425 1997 catalog_manager.cc:5484] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.573925 2538 consensus_queue.cc:237] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:45.576320 2147 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.576525 2406 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:45.576987 2472 consensus_peers.cc:485] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cb2dab6f4684bb18bddc9edba1d6517. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.577042 5976 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.577913 6905 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 14:00:45.581046 5976 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:45.583106 2147 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:45.587040 6545 raft_consensus.cc:1050] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:45.587853 6545 consensus_queue.cc:237] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:45.590140 2000 catalog_manager.cc:5484] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.594732 2406 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:45.598104 2274 raft_consensus.cc:1261] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.598146 2538 raft_consensus.cc:1261] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.598826 6544 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.599593 6544 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.599884 5164 raft_consensus.cc:1261] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.601825 6544 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.609006 6544 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.609638 2538 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.617693 2000 catalog_manager.cc:5484] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.620748 5164 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.625589 2274 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.628718 2406 consensus_queue.cc:237] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:45.631212 2538 raft_consensus.cc:1261] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.632686 5976 raft_consensus.cc:1050] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:45.632700 6544 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.633443 5976 consensus_queue.cc:237] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:45.633802 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:45.638888 5164 raft_consensus.cc:1261] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.639621 6544 consensus_queue.cc:1035] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.645376 6857 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:45.645750 2408 raft_consensus.cc:1261] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.645933 4900 raft_consensus.cc:1261] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.646987 6715 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.647596 6715 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.648484 6544 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.650763 6862 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:45.651178 6858 tablet.cc:1722] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:45.652472 2275 consensus_queue.cc:237] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:45.655268 6653 consensus_queue.cc:1035] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.658790 5164 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.661322 2538 raft_consensus.cc:2932] T e5f7a0845388479b9d5f5f12d776c978 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:45.662748 1999 catalog_manager.cc:5484] T e5f7a0845388479b9d5f5f12d776c978 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:45.664069 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5f7a0845388479b9d5f5f12d776c978 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:45.666839 2004 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=0651835ef96943f69dbd61ba74cc3ab0] alter complete (version 3)
I20240501 14:00:45.671499 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet e5f7a0845388479b9d5f5f12d776c978 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:45.679436 6924 tablet_replica.cc:330] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:45.681728 6653 consensus_queue.cc:1035] T 913ae5fda9e842e1b3a6d7dc418a5bfd P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.684166 2274 raft_consensus.cc:1261] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:45.684983 6905 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:45.689397 6715 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.694794 6924 raft_consensus.cc:2229] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.698213 2406 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.700093 6924 raft_consensus.cc:2258] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.700351 2143 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.701344 6878 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.708184 6924 ts_tablet_manager.cc:1875] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:45.708803 2006 catalog_manager.cc:5484] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:45.710377 2538 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:45.697482 4901 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.711028 6878 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:45.717906 6878 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
W20240501 14:00:45.722110 2211 consensus_peers.cc:485] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65306dbef405443fb4c7188ccb762b83. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.723160 2533 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:45.724532 2006 catalog_manager.cc:5484] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.734884 2147 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:45.743927 6924 ts_tablet_manager.cc:1888] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:45.750828 6924 log.cc:1196] T e5f7a0845388479b9d5f5f12d776c978 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5f7a0845388479b9d5f5f12d776c978
I20240501 14:00:45.751633 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet e5f7a0845388479b9d5f5f12d776c978 (table rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]) successfully deleted
I20240501 14:00:45.751997 2274 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.755941 2533 consensus_queue.cc:237] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:45.758450 4901 raft_consensus.cc:1261] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.759019 5976 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.760042 2274 raft_consensus.cc:1261] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:45.760610 5976 consensus_queue.cc:1035] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:45.762189 6928 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.763406 2274 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.765308 4901 raft_consensus.cc:2932] T 304b465c0a99456790bfc8f262b61ecd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.765722 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 304b465c0a99456790bfc8f262b61ecd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:45.769968 2001 catalog_manager.cc:5484] T 304b465c0a99456790bfc8f262b61ecd P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:45.771209 2006 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:45.771421 2006 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:45.774250 6800 consensus_queue.cc:1035] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.775228 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 304b465c0a99456790bfc8f262b61ecd with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:45.785311 6821 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.790174 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 948b150d59024a58b813b9353d665394
I20240501 14:00:45.791105 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 948b150d59024a58b813b9353d665394 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:45.791724 6941 tablet_replica.cc:330] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:45.792114 6941 raft_consensus.cc:2229] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.794149 6941 raft_consensus.cc:2258] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.795326 6941 ts_tablet_manager.cc:1875] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.797294 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet 948b150d59024a58b813b9353d665394 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:45.797375 6686 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:45.797751 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 948b150d59024a58b813b9353d665394 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:45 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:45.804257 6941 ts_tablet_manager.cc:1888] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.805161 6945 tablet_replica.cc:330] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:45.805452 6945 raft_consensus.cc:2229] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.805732 6945 raft_consensus.cc:2258] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.806717 6945 ts_tablet_manager.cc:1875] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.808844 6941 log.cc:1196] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/948b150d59024a58b813b9353d665394
I20240501 14:00:45.809410 6941 ts_tablet_manager.cc:1909] T 948b150d59024a58b813b9353d665394 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:45.810570 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table_new_name [id=0651835ef96943f69dbd61ba74cc3ab0]) successfully deleted
I20240501 14:00:45.817426 6944 tablet_replica.cc:330] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:45.819129 6938 tablet_replica.cc:330] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:45.819404 6945 ts_tablet_manager.cc:1888] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.819617 6945 log.cc:1196] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/948b150d59024a58b813b9353d665394
I20240501 14:00:45.819962 6944 raft_consensus.cc:2229] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.820116 6945 ts_tablet_manager.cc:1909] T 948b150d59024a58b813b9353d665394 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:45.821252 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table_new_name [id=0651835ef96943f69dbd61ba74cc3ab0]) successfully deleted
I20240501 14:00:45.827082 6944 raft_consensus.cc:2258] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.825958 6938 raft_consensus.cc:2229] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.827612 6938 raft_consensus.cc:2258] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.828733 6944 ts_tablet_manager.cc:1875] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:45.831104 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:45.836119 6944 ts_tablet_manager.cc:1888] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:45.836393 6944 log.cc:1196] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/948b150d59024a58b813b9353d665394
I20240501 14:00:45.842631 2516 tablet_service.cc:1467] Processing CreateTablet for tablet b147c5f61a574c7d8a334b5f67ce732d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.843106 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b147c5f61a574c7d8a334b5f67ce732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.842752 4879 tablet_service.cc:1467] Processing CreateTablet for tablet e9a9e1d4df0a48f59b03349380cb6608 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.843487 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a9e1d4df0a48f59b03349380cb6608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.843834 2125 tablet_service.cc:1467] Processing CreateTablet for tablet b9ccb0f16d8044da9f3f353ffbea5ce9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.843948 5013 tablet_service.cc:1467] Processing CreateTablet for tablet e9a9e1d4df0a48f59b03349380cb6608 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.844216 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ccb0f16d8044da9f3f353ffbea5ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.844342 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a9e1d4df0a48f59b03349380cb6608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.846909 5012 tablet_service.cc:1467] Processing CreateTablet for tablet b147c5f61a574c7d8a334b5f67ce732d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.846966 5142 tablet_service.cc:1467] Processing CreateTablet for tablet e9a9e1d4df0a48f59b03349380cb6608 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.847303 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b147c5f61a574c7d8a334b5f67ce732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.847362 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a9e1d4df0a48f59b03349380cb6608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.837743 6947 consensus_queue.cc:1035] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240501 14:00:45.850458 6944 ts_tablet_manager.cc:1909] T 948b150d59024a58b813b9353d665394 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:45.850876 5014 tablet_service.cc:1467] Processing CreateTablet for tablet b9ccb0f16d8044da9f3f353ffbea5ce9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.851269 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ccb0f16d8044da9f3f353ffbea5ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.851675 5274 tablet_service.cc:1467] Processing CreateTablet for tablet b147c5f61a574c7d8a334b5f67ce732d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.852119 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b147c5f61a574c7d8a334b5f67ce732d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.855296 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 948b150d59024a58b813b9353d665394 (table rebalancer_dynamic_table_new_name [id=0651835ef96943f69dbd61ba74cc3ab0]) successfully deleted
I20240501 14:00:45.855695 4881 tablet_service.cc:1467] Processing CreateTablet for tablet b9ccb0f16d8044da9f3f353ffbea5ce9 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:45.856139 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ccb0f16d8044da9f3f353ffbea5ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.866835 6949 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:45.869187 6949 tablet_bootstrap.cc:654] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.870347 5161 consensus_queue.cc:237] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:00:45.877467 6951 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:45.878165 6949 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:45.878413 6949 ts_tablet_manager.cc:1367] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 14:00:45.879858 6949 raft_consensus.cc:348] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.881369 6949 raft_consensus.cc:374] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.881522 6949 raft_consensus.cc:729] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.881853 6949 consensus_queue.cc:260] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.882797 6951 tablet_bootstrap.cc:654] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.883545 6949 ts_tablet_manager.cc:1398] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:00:45.883889 6949 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:45.885664 6949 tablet_bootstrap.cc:654] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.891844 6951 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:45.892114 6951 ts_tablet_manager.cc:1367] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240501 14:00:45.893036 6951 raft_consensus.cc:348] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.893368 6951 raft_consensus.cc:374] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.893497 6951 raft_consensus.cc:729] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.894275 6951 consensus_queue.cc:260] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.895170 6951 ts_tablet_manager.cc:1398] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:45.895603 6951 tablet_bootstrap.cc:492] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:45.895709 6949 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:45.895921 6949 ts_tablet_manager.cc:1367] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240501 14:00:45.896718 6949 raft_consensus.cc:348] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.896989 6949 raft_consensus.cc:374] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.897121 6949 raft_consensus.cc:729] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.897266 6951 tablet_bootstrap.cc:654] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.897424 6949 consensus_queue.cc:260] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:45.901587 6949 ts_tablet_manager.cc:1398] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:00:45.903913 6959 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:45.906298 6959 tablet_bootstrap.cc:654] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.904448 6958 ts_tablet_manager.cc:902] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:45.908655 6951 tablet_bootstrap.cc:492] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:45.908936 6951 ts_tablet_manager.cc:1367] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240501 14:00:45.909808 6951 raft_consensus.cc:348] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.910130 6951 raft_consensus.cc:374] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.910244 6951 raft_consensus.cc:729] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.910931 6951 consensus_queue.cc:260] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.911731 6951 ts_tablet_manager.cc:1398] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:45.912073 6951 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:45.913167 6938 ts_tablet_manager.cc:1875] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:45.913764 6951 tablet_bootstrap.cc:654] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.916914 6959 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:45.917181 6959 ts_tablet_manager.cc:1367] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240501 14:00:45.918059 6951 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:45.918313 6951 ts_tablet_manager.cc:1367] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:00:45.918227 6959 raft_consensus.cc:348] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.919399 6959 raft_consensus.cc:374] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.919665 6959 raft_consensus.cc:729] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.919935 6957 tablet_bootstrap.cc:492] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:45.919997 6951 raft_consensus.cc:348] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.920253 6951 raft_consensus.cc:374] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.920217 6959 consensus_queue.cc:260] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.920425 6951 raft_consensus.cc:729] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.920732 6951 consensus_queue.cc:260] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:45.921476 6951 ts_tablet_manager.cc:1398] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:45.920470 6961 tablet_bootstrap.cc:492] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:45.922353 6959 ts_tablet_manager.cc:1398] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:45.922560 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:45.932197 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 88c8d6774bd944ceb98eb1769a619eb6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21]), partition=
I20240501 14:00:45.932493 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 88c8d6774bd944ceb98eb1769a619eb6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21]), partition=
I20240501 14:00:45.932687 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c8d6774bd944ceb98eb1769a619eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.932883 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c8d6774bd944ceb98eb1769a619eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.933205 6957 tablet_bootstrap.cc:654] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.934070 6961 tablet_bootstrap.cc:654] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.934813 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 88c8d6774bd944ceb98eb1769a619eb6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21]), partition=
I20240501 14:00:45.935156 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c8d6774bd944ceb98eb1769a619eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.937752 6961 tablet_bootstrap.cc:492] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:45.939884 6949 tablet_bootstrap.cc:492] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:45.941139 6949 tablet_bootstrap.cc:654] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.942554 6961 ts_tablet_manager.cc:1367] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240501 14:00:45.944166 6961 raft_consensus.cc:348] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.944412 6961 raft_consensus.cc:374] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.946835 6961 raft_consensus.cc:729] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.947471 6961 consensus_queue.cc:260] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.950235 6957 tablet_bootstrap.cc:492] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:45.950809 6957 ts_tablet_manager.cc:1367] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240501 14:00:45.951659 6957 raft_consensus.cc:348] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.951992 6957 raft_consensus.cc:374] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.952117 6957 raft_consensus.cc:729] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.952435 6957 consensus_queue.cc:260] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:45.956657 6949 tablet_bootstrap.cc:492] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:45.956835 6949 ts_tablet_manager.cc:1367] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240501 14:00:45.957670 6949 raft_consensus.cc:348] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.957871 6949 raft_consensus.cc:374] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.957955 6949 raft_consensus.cc:729] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.958137 6957 ts_tablet_manager.cc:1398] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:45.958211 6949 consensus_queue.cc:260] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
W20240501 14:00:45.958362 5097 consensus_peers.cc:485] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13d8ec926b7a46568470e824763569e6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:45.958881 6949 ts_tablet_manager.cc:1398] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:45.959718 6961 ts_tablet_manager.cc:1398] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240501 14:00:45.964707 2274 raft_consensus.cc:1261] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:45.965381 6686 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:45.966485 5293 raft_consensus.cc:1261] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:45.968443 6686 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:45.972812 6686 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:00:45.973811 6959 tablet_bootstrap.cc:492] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:45.976116 6959 tablet_bootstrap.cc:654] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.975970 2274 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:00:45.982525 6959 tablet_bootstrap.cc:492] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:45.984300 6959 ts_tablet_manager.cc:1367] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240501 14:00:45.985181 6959 raft_consensus.cc:348] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.985492 6959 raft_consensus.cc:374] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.985590 6959 raft_consensus.cc:729] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.985877 6959 consensus_queue.cc:260] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:45.986617 6959 ts_tablet_manager.cc:1398] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:45.988226 6970 ts_tablet_manager.cc:902] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:45.988248 6938 ts_tablet_manager.cc:1888] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:45.989969 6950 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:45.993744 2000 catalog_manager.cc:5484] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:45.997181 6961 tablet_bootstrap.cc:492] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:45.998831 6950 tablet_bootstrap.cc:654] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.998855 6961 tablet_bootstrap.cc:654] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.999572 6938 log.cc:1196] T 304b465c0a99456790bfc8f262b61ecd P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/304b465c0a99456790bfc8f262b61ecd
I20240501 14:00:46.000856 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 304b465c0a99456790bfc8f262b61ecd (table rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]) successfully deleted
I20240501 14:00:46.003188 5293 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:00:46.010203 6970 tablet_copy_client.cc:308] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:46.011272 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet dcb6951344ad49b3aee27205e898c347 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:53689)
I20240501 14:00:46.011451 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet dcb6951344ad49b3aee27205e898c347 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:53689: session id = 5a4b9a96e9734baa8a03d6056bfa934c-dcb6951344ad49b3aee27205e898c347
I20240501 14:00:46.013743 2428 tablet_copy_source_session.cc:205] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.013872 6961 tablet_bootstrap.cc:492] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:46.014041 6961 ts_tablet_manager.cc:1367] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.001s
I20240501 14:00:46.015390 6970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb6951344ad49b3aee27205e898c347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.015450 6961 raft_consensus.cc:348] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.015749 6961 raft_consensus.cc:374] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.015877 6961 raft_consensus.cc:729] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.018314 6961 consensus_queue.cc:260] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.019556 6950 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:46.019723 6950 ts_tablet_manager.cc:1367] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240501 14:00:46.020509 6950 raft_consensus.cc:348] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:46.020726 6950 raft_consensus.cc:374] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.020817 6950 raft_consensus.cc:729] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.021036 6950 consensus_queue.cc:260] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:46.021590 6950 ts_tablet_manager.cc:1398] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:46.024528 6958 tablet_copy_client.cc:308] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:00:46.026309 6961 ts_tablet_manager.cc:1398] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:00:46.027894 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 73940622ad734b88af31e9fa0c1000e5 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:38303)
I20240501 14:00:46.028093 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 73940622ad734b88af31e9fa0c1000e5 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:38303: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-73940622ad734b88af31e9fa0c1000e5
I20240501 14:00:46.030689 2298 tablet_copy_source_session.cc:205] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.032819 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73940622ad734b88af31e9fa0c1000e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.033425 6970 tablet_copy_client.cc:783] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.033771 6970 tablet_copy_client.cc:647] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.036216 6970 tablet_copy_client.cc:523] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.038365 6927 raft_consensus.cc:482] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.038537 6927 raft_consensus.cc:504] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.041270 5965 raft_consensus.cc:482] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.041939 5965 raft_consensus.cc:504] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.042651 5965 leader_election.cc:288] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:46.043280 6958 tablet_copy_client.cc:783] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.043649 6958 tablet_copy_client.cc:647] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.043634 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a9e1d4df0a48f59b03349380cb6608" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:46.043864 4901 raft_consensus.cc:2454] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:46.044677 5098 leader_election.cc:302] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:46.045224 6821 raft_consensus.cc:2781] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.045344 6821 raft_consensus.cc:482] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.045435 6821 raft_consensus.cc:3037] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.048142 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147c5f61a574c7d8a334b5f67ce732d" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:46.048172 6821 raft_consensus.cc:504] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.048362 5293 raft_consensus.cc:2454] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:46.048403 6927 leader_election.cc:288] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:46.049203 4969 leader_election.cc:302] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 6f81f803ec7746ccb502c59b7cbf8eb2; no voters:
I20240501 14:00:46.049279 6821 leader_election.cc:288] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:46.049747 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147c5f61a574c7d8a334b5f67ce732d" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:46.049976 2538 raft_consensus.cc:2454] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:46.050801 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a9e1d4df0a48f59b03349380cb6608" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:46.050875 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a9e1d4df0a48f59b03349380cb6608" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:46.051023 5034 raft_consensus.cc:3037] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.051748 6970 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:46.052487 6927 raft_consensus.cc:2781] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.052608 6927 raft_consensus.cc:482] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.052592 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a9e1d4df0a48f59b03349380cb6608" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:46.052716 6927 raft_consensus.cc:3037] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.052837 4900 raft_consensus.cc:3037] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.053794 5034 raft_consensus.cc:2454] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:46.055030 5099 leader_election.cc:302] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:46.055953 4900 raft_consensus.cc:2454] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:46.056993 6927 raft_consensus.cc:504] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.057695 6927 leader_election.cc:288] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:46.057935 6821 raft_consensus.cc:2781] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:46.058077 6821 raft_consensus.cc:686] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:46.058368 6821 consensus_queue.cc:237] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.059342 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147c5f61a574c7d8a334b5f67ce732d" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:46.059592 5293 raft_consensus.cc:3037] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.059556 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147c5f61a574c7d8a334b5f67ce732d" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:46.059799 2538 raft_consensus.cc:3037] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.062237 5293 raft_consensus.cc:2454] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:46.062675 4969 leader_election.cc:302] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 6f81f803ec7746ccb502c59b7cbf8eb2; no voters:
I20240501 14:00:46.063891 2538 raft_consensus.cc:2454] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:46.064643 6927 raft_consensus.cc:2781] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:46.065255 6927 raft_consensus.cc:686] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:46.065627 6927 consensus_queue.cc:237] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.074975 6991 ts_tablet_manager.cc:902] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:46.075819 6958 tablet_copy_client.cc:523] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.080955 5034 consensus_queue.cc:237] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:46.081526 6958 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:46.084877 2006 catalog_manager.cc:5484] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:46.096557 6987 ts_tablet_manager.cc:902] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
W20240501 14:00:46.099551 4967 consensus_peers.cc:485] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5086890bd254edebdf12812ff5c3728. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:46.104775 4900 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.105593 6929 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:46.110985 6948 raft_consensus.cc:482] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.111171 6948 raft_consensus.cc:504] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:46.112566 6948 leader_election.cc:288] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:46.113159 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccb0f16d8044da9f3f353ffbea5ce9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:46.113322 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccb0f16d8044da9f3f353ffbea5ce9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:46.113408 2147 raft_consensus.cc:2454] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:46.113531 5032 raft_consensus.cc:2454] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:46.113906 4836 leader_election.cc:302] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:46.114187 6948 raft_consensus.cc:2781] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.114301 6948 raft_consensus.cc:482] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.114403 6948 raft_consensus.cc:3037] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.117175 6948 raft_consensus.cc:504] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:46.117861 6948 leader_election.cc:288] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:46.118363 2147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccb0f16d8044da9f3f353ffbea5ce9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:46.118582 2147 raft_consensus.cc:3037] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.118955 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccb0f16d8044da9f3f353ffbea5ce9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:46.119146 5032 raft_consensus.cc:3037] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.121301 2147 raft_consensus.cc:2454] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:46.121762 5032 raft_consensus.cc:2454] T b9ccb0f16d8044da9f3f353ffbea5ce9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:46.121927 4834 leader_election.cc:302] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:46.122705 6821 raft_consensus.cc:482] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.122880 6821 raft_consensus.cc:504] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.123622 6821 leader_election.cc:288] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:46.123714 6970 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.124033 6948 raft_consensus.cc:2781] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:46.124118 6970 tablet_bootstrap.cc:492] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:00:46.124379 6970 ts_tablet_manager.cc:1367] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.073s user 0.046s sys 0.004s
I20240501 14:00:46.125106 6948 raft_consensus.cc:686] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:46.125463 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c8d6774bd944ceb98eb1769a619eb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:46.125074 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c8d6774bd944ceb98eb1769a619eb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:46.125715 2538 raft_consensus.cc:2454] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:46.125698 6970 raft_consensus.cc:348] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:46.126029 6970 raft_consensus.cc:729] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:00:46.126143 5097 leader_election.cc:302] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:46.125838 4900 raft_consensus.cc:2454] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:46.126520 6821 raft_consensus.cc:2781] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.126513 6970 consensus_queue.cc:260] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:00:46.127110 6970 ts_tablet_manager.cc:1398] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:46.127154 6821 raft_consensus.cc:482] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.127280 6821 raft_consensus.cc:3037] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.128341 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-dcb6951344ad49b3aee27205e898c347 received from {username='slave'} at 127.28.17.8:53689
I20240501 14:00:46.128475 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-dcb6951344ad49b3aee27205e898c347 on tablet dcb6951344ad49b3aee27205e898c347 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:46.125605 6948 consensus_queue.cc:237] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:46.130117 6821 raft_consensus.cc:504] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.131671 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c8d6774bd944ceb98eb1769a619eb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:46.131922 2533 raft_consensus.cc:3037] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.134704 6991 tablet_copy_client.cc:308] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:46.134477 2533 raft_consensus.cc:2454] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:46.135474 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 1cb2dab6f4684bb18bddc9edba1d6517 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:50935)
I20240501 14:00:46.135576 5097 leader_election.cc:302] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:46.135620 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 1cb2dab6f4684bb18bddc9edba1d6517 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:50935: session id = f8c60f2bbac541dcba4424ab726f9ad9-1cb2dab6f4684bb18bddc9edba1d6517
I20240501 14:00:46.136062 5965 raft_consensus.cc:2781] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:46.136202 5965 raft_consensus.cc:686] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:46.137001 5965 consensus_queue.cc:237] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.137905 2558 tablet_copy_source_session.cc:205] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.138314 2147 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.139219 6821 leader_election.cc:288] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:46.139391 6991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2dab6f4684bb18bddc9edba1d6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.139411 6929 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:46.139681 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c8d6774bd944ceb98eb1769a619eb6" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:46.139950 4900 raft_consensus.cc:3037] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.144626 4900 raft_consensus.cc:2454] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:46.148126 6991 tablet_copy_client.cc:783] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.147763 6929 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.148519 6991 tablet_copy_client.cc:647] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.150178 6991 tablet_copy_client.cc:523] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.152338 2006 catalog_manager.cc:5484] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:46.158198 4900 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.163242 2146 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.172796 2006 catalog_manager.cc:5484] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.173473 6991 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:46.203033 1999 catalog_manager.cc:5484] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:46.225548 1999 catalog_manager.cc:5484] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:46.228642 6987 tablet_copy_client.cc:308] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:00:46.229715 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 44a4e92b144f47b5800506f4bc3eadb6 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:54429)
I20240501 14:00:46.229965 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 44a4e92b144f47b5800506f4bc3eadb6 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:54429: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-44a4e92b144f47b5800506f4bc3eadb6
I20240501 14:00:46.233644 5184 tablet_copy_source_session.cc:205] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.243175 6987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4e92b144f47b5800506f4bc3eadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.243912 6958 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.244410 6958 tablet_bootstrap.cc:492] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:46.245640 6958 ts_tablet_manager.cc:1367] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.164s user 0.044s sys 0.012s
I20240501 14:00:46.246364 6958 raft_consensus.cc:348] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:46.246652 6958 raft_consensus.cc:729] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:46.246882 6958 consensus_queue.cc:260] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:46.250190 6991 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.250532 6987 tablet_copy_client.cc:783] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.250661 6991 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:46.250874 6987 tablet_copy_client.cc:647] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.250941 6991 ts_tablet_manager.cc:1367] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.078s user 0.037s sys 0.009s
I20240501 14:00:46.251767 6991 raft_consensus.cc:348] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:46.252002 6991 raft_consensus.cc:729] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:46.252203 6991 consensus_queue.cc:260] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:46.252806 6958 ts_tablet_manager.cc:1398] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:00:46.252916 6991 ts_tablet_manager.cc:1398] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:46.253446 6987 tablet_copy_client.cc:523] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.257972 6987 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:46.259748 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-1cb2dab6f4684bb18bddc9edba1d6517 received from {username='slave'} at 127.28.17.5:50935
I20240501 14:00:46.259771 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-73940622ad734b88af31e9fa0c1000e5 received from {username='slave'} at 127.28.17.7:38303
I20240501 14:00:46.259909 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-1cb2dab6f4684bb18bddc9edba1d6517 on tablet 1cb2dab6f4684bb18bddc9edba1d6517 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:46.259910 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-73940622ad734b88af31e9fa0c1000e5 on tablet 73940622ad734b88af31e9fa0c1000e5 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:46.262071 2275 consensus_queue.cc:237] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:46.263491 6958 ts_tablet_manager.cc:902] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:46.268791 2146 raft_consensus.cc:1261] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:46.269265 2538 raft_consensus.cc:1261] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:46.269469 6764 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:46.269800 2211 consensus_peers.cc:485] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 238aa15a2cd74ff69e17a2934b32b70d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:46.270154 6878 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:46.273649 6764 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:46.274132 2146 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:46.279285 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:46.282428 2000 catalog_manager.cc:5484] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:46.286355 6987 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.286898 6987 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:46.287225 6987 ts_tablet_manager.cc:1367] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.003s
I20240501 14:00:46.288077 6987 raft_consensus.cc:348] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:46.288322 6987 raft_consensus.cc:729] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:46.288337 4900 raft_consensus.cc:1261] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.288561 6987 consensus_queue.cc:260] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:46.289165 6987 ts_tablet_manager.cc:1398] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:46.289124 6821 consensus_queue.cc:1035] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.289775 2538 raft_consensus.cc:1261] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.290480 6821 consensus_queue.cc:1035] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.292305 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-44a4e92b144f47b5800506f4bc3eadb6 received from {username='slave'} at 127.28.17.6:54429
I20240501 14:00:46.292488 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-44a4e92b144f47b5800506f4bc3eadb6 on tablet 44a4e92b144f47b5800506f4bc3eadb6 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:46.293535 2533 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:00:46.296666 1999 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 88c8d6774bd944ceb98eb1769a619eb6 (table rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21]). Expected version 1 got 0
I20240501 14:00:46.297195 7026 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:46.298134 7026 tablet.cc:1620] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:46.308015 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21] alter complete (version 1)
I20240501 14:00:46.318136 7025 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:46.318778 7025 tablet.cc:1620] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:46.320120 6958 tablet_copy_client.cc:308] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:00:46.321055 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 65306dbef405443fb4c7188ccb762b83 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:38303)
I20240501 14:00:46.321437 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 65306dbef405443fb4c7188ccb762b83 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:38303: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-65306dbef405443fb4c7188ccb762b83
I20240501 14:00:46.322577 7033 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:46.323427 7033 tablet.cc:1620] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:46.324180 2298 tablet_copy_source_session.cc:205] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.324728 1999 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet 88c8d6774bd944ceb98eb1769a619eb6 (table rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21]). Expected version 1 got 0
I20240501 14:00:46.325532 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65306dbef405443fb4c7188ccb762b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.328452 2000 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet 88c8d6774bd944ceb98eb1769a619eb6 (table rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21]). Expected version 1 got 0
I20240501 14:00:46.333887 6958 tablet_copy_client.cc:783] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.334199 6958 tablet_copy_client.cc:647] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.335958 6958 tablet_copy_client.cc:523] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.339315 6958 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:46.345592 4900 consensus_queue.cc:237] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
W20240501 14:00:46.360560 4834 consensus_peers.cc:485] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaaae937ea4d44068d3ad5b9024b9421. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:46.361721 5164 raft_consensus.cc:1261] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.362393 6947 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:46.363099 2146 raft_consensus.cc:1261] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:46.364537 6947 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:46.369186 6947 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.372478 5164 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.381148 1999 catalog_manager.cc:5484] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.391048 2146 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:00:46.399350 6958 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.399839 6958 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:46.400110 6958 ts_tablet_manager.cc:1367] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.061s user 0.042s sys 0.007s
I20240501 14:00:46.400990 6958 raft_consensus.cc:348] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:46.401319 6958 raft_consensus.cc:729] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:46.401547 6958 consensus_queue.cc:260] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:46.402379 6958 ts_tablet_manager.cc:1398] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:46.403228 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-65306dbef405443fb4c7188ccb762b83 received from {username='slave'} at 127.28.17.7:38303
I20240501 14:00:46.403403 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-65306dbef405443fb4c7188ccb762b83 on tablet 65306dbef405443fb4c7188ccb762b83 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:46.411952 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:46.421192 7025 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:46.421663 7025 tablet.cc:1620] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:46.423493 7026 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:46.424026 7026 tablet.cc:1620] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:46.427894 7033 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:46.428418 7033 tablet.cc:1620] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:46.432018 5164 raft_consensus.cc:1203] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:46.439761 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=5ca311bb50d54cc6905d0b3aa96eed21] alter complete (version 2)
I20240501 14:00:46.447839 5293 raft_consensus.cc:1203] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:46.462451 6927 consensus_queue.cc:1035] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.470818 6653 consensus_queue.cc:1035] T b147c5f61a574c7d8a334b5f67ce732d P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.502279 7062 ts_tablet_manager.cc:902] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:00:46.507524 5099 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.8:34221, user_credentials={real_user=slave}} blocked reactor thread for 86660.6us
I20240501 14:00:46.509138 7063 ts_tablet_manager.cc:902] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:46.512898 5034 raft_consensus.cc:1203] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:46.526945 7062 tablet_copy_client.cc:308] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.6:46037
I20240501 14:00:46.527252 7051 consensus_queue.cc:1035] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 14:00:46.527776 5054 tablet_copy_service.cc:140] P 6f81f803ec7746ccb502c59b7cbf8eb2: Received BeginTabletCopySession request for tablet d5086890bd254edebdf12812ff5c3728 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:48677)
I20240501 14:00:46.527926 5054 tablet_copy_service.cc:161] P 6f81f803ec7746ccb502c59b7cbf8eb2: Beginning new tablet copy session on tablet d5086890bd254edebdf12812ff5c3728 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:48677: session id = a64f34785a7b44fa8070b0452a4e0ddd-d5086890bd254edebdf12812ff5c3728
I20240501 14:00:46.530937 5054 tablet_copy_source_session.cc:205] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.537309 7063 tablet_copy_client.cc:308] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:00:46.537235 7051 consensus_queue.cc:1035] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.538800 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 13d8ec926b7a46568470e824763569e6 from peer d1a414d381f44829a2510cb76d481887 ({username='slave'} at 127.28.17.1:40345)
I20240501 14:00:46.538995 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 13d8ec926b7a46568470e824763569e6 from peer d1a414d381f44829a2510cb76d481887 at {username='slave'} at 127.28.17.1:40345: session id = d1a414d381f44829a2510cb76d481887-13d8ec926b7a46568470e824763569e6
I20240501 14:00:46.541687 5184 tablet_copy_source_session.cc:205] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.543066 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:46.543875 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d8ec926b7a46568470e824763569e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.551288 7063 tablet_copy_client.cc:783] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.552048 7063 tablet_copy_client.cc:647] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.552404 7025 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:46.558548 7033 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:46.558568 7026 tablet.cc:1722] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:46.561201 7063 tablet_copy_client.cc:523] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.548509 7062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5086890bd254edebdf12812ff5c3728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.570253 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=5ca311bb50d54cc6905d0b3aa96eed21] alter complete (version 3)
I20240501 14:00:46.575114 7063 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:46.577518 7062 tablet_copy_client.cc:783] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.578361 7062 tablet_copy_client.cc:647] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.580157 7062 tablet_copy_client.cc:523] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.586491 7062 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:46.593932 7063 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.594444 7063 tablet_bootstrap.cc:492] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: Bootstrap complete.
I20240501 14:00:46.594730 7063 ts_tablet_manager.cc:1367] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.002s
I20240501 14:00:46.595574 7063 raft_consensus.cc:348] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:00:46.595836 7063 raft_consensus.cc:729] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: LEARNER
I20240501 14:00:46.596061 7063 consensus_queue.cc:260] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:00:46.596913 7063 ts_tablet_manager.cc:1398] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:46.601123 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session d1a414d381f44829a2510cb76d481887-13d8ec926b7a46568470e824763569e6 received from {username='slave'} at 127.28.17.1:40345
I20240501 14:00:46.601238 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session d1a414d381f44829a2510cb76d481887-13d8ec926b7a46568470e824763569e6 on tablet 13d8ec926b7a46568470e824763569e6 with peer d1a414d381f44829a2510cb76d481887
I20240501 14:00:46.602574 7062 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.603006 7062 tablet_bootstrap.cc:492] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:00:46.603744 7062 ts_tablet_manager.cc:1367] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240501 14:00:46.605125 7062 raft_consensus.cc:348] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:46.605403 7062 raft_consensus.cc:729] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:00:46.605620 7062 consensus_queue.cc:260] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:46.606130 7062 ts_tablet_manager.cc:1398] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:46.607399 5054 tablet_copy_service.cc:342] P 6f81f803ec7746ccb502c59b7cbf8eb2: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-d5086890bd254edebdf12812ff5c3728 received from {username='slave'} at 127.28.17.3:48677
I20240501 14:00:46.607537 5054 tablet_copy_service.cc:433] P 6f81f803ec7746ccb502c59b7cbf8eb2: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-d5086890bd254edebdf12812ff5c3728 on tablet d5086890bd254edebdf12812ff5c3728 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:46.630445 6821 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.641683 6821 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:46.643530 4901 raft_consensus.cc:1203] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:46.674847 2006 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:46.675132 2006 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:46.680187 2006 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 88c8d6774bd944ceb98eb1769a619eb6
I20240501 14:00:46.682807 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c8d6774bd944ceb98eb1769a619eb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:46.683596 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c8d6774bd944ceb98eb1769a619eb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:46.684068 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c8d6774bd944ceb98eb1769a619eb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:46 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:46.696205 7106 tablet_replica.cc:330] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:46.696637 7106 raft_consensus.cc:2229] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.696955 7106 raft_consensus.cc:2258] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.698596 7106 ts_tablet_manager.cc:1875] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.710557 7106 ts_tablet_manager.cc:1888] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:46.710847 7106 log.cc:1196] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/88c8d6774bd944ceb98eb1769a619eb6
I20240501 14:00:46.711423 7106 ts_tablet_manager.cc:1909] T 88c8d6774bd944ceb98eb1769a619eb6 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:46.712630 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 88c8d6774bd944ceb98eb1769a619eb6 (table rebalancer_dynamic_table_new_name [id=5ca311bb50d54cc6905d0b3aa96eed21]) successfully deleted
I20240501 14:00:46.714395 7104 tablet_replica.cc:330] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:46.714648 7104 raft_consensus.cc:2229] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:46.714864 7104 raft_consensus.cc:2258] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.715291 7105 tablet_replica.cc:330] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:46.715497 7105 raft_consensus.cc:2229] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:46.715809 7105 raft_consensus.cc:2258] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:46.716603 7104 ts_tablet_manager.cc:1875] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.717145 7105 ts_tablet_manager.cc:1875] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:46.726752 7105 ts_tablet_manager.cc:1888] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:46.726982 7105 log.cc:1196] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/88c8d6774bd944ceb98eb1769a619eb6
I20240501 14:00:46.727483 7105 ts_tablet_manager.cc:1909] T 88c8d6774bd944ceb98eb1769a619eb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:46.728750 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 88c8d6774bd944ceb98eb1769a619eb6 (table rebalancer_dynamic_table_new_name [id=5ca311bb50d54cc6905d0b3aa96eed21]) successfully deleted
I20240501 14:00:46.733850 7104 ts_tablet_manager.cc:1888] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:46.734165 7104 log.cc:1196] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/88c8d6774bd944ceb98eb1769a619eb6
I20240501 14:00:46.734716 7104 ts_tablet_manager.cc:1909] T 88c8d6774bd944ceb98eb1769a619eb6 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:46.735896 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 88c8d6774bd944ceb98eb1769a619eb6 (table rebalancer_dynamic_table_new_name [id=5ca311bb50d54cc6905d0b3aa96eed21]) successfully deleted
I20240501 14:00:46.737635 7062 ts_tablet_manager.cc:902] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:46.739923 7062 tablet_copy_client.cc:308] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:00:46.740974 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet aaaae937ea4d44068d3ad5b9024b9421 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:36677)
I20240501 14:00:46.741191 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet aaaae937ea4d44068d3ad5b9024b9421 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:36677: session id = a64f34785a7b44fa8070b0452a4e0ddd-aaaae937ea4d44068d3ad5b9024b9421
I20240501 14:00:46.745055 4921 tablet_copy_source_session.cc:205] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.746919 7062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaaae937ea4d44068d3ad5b9024b9421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.752951 7062 tablet_copy_client.cc:783] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.753234 7062 tablet_copy_client.cc:647] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.753652 2006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:46.756544 7062 tablet_copy_client.cc:523] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.761070 7062 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:46.767105 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 3dff7dc57d1e4b0493778a54bc2c2c5b (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.767563 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dff7dc57d1e4b0493778a54bc2c2c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.767668 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 52eb73b91ed441199eade33e66b84b39 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.767899 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 52eb73b91ed441199eade33e66b84b39 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.768059 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52eb73b91ed441199eade33e66b84b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.768329 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52eb73b91ed441199eade33e66b84b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.773705 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 3dff7dc57d1e4b0493778a54bc2c2c5b (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.773696 5014 tablet_service.cc:1467] Processing CreateTablet for tablet 3dff7dc57d1e4b0493778a54bc2c2c5b (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.774164 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dff7dc57d1e4b0493778a54bc2c2c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.774199 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dff7dc57d1e4b0493778a54bc2c2c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.774871 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 3697989873e94a6fad3835e0ebb36fce (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.774922 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 52eb73b91ed441199eade33e66b84b39 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.776305 7062 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:46.776736 7062 tablet_bootstrap.cc:492] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:00:46.776986 7062 ts_tablet_manager.cc:1367] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240501 14:00:46.777751 7062 raft_consensus.cc:348] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:46.777987 7062 raft_consensus.cc:729] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:00:46.778211 7062 consensus_queue.cc:260] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:00:46.778801 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3697989873e94a6fad3835e0ebb36fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.778791 7062 ts_tablet_manager.cc:1398] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:46.785514 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-aaaae937ea4d44068d3ad5b9024b9421 received from {username='slave'} at 127.28.17.3:36677
I20240501 14:00:46.785748 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-aaaae937ea4d44068d3ad5b9024b9421 on tablet aaaae937ea4d44068d3ad5b9024b9421 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:00:46.789553 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52eb73b91ed441199eade33e66b84b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.795868 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:46.802031 5164 raft_consensus.cc:1203] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:46.805352 7124 tablet_bootstrap.cc:492] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:46.811111 7124 tablet_bootstrap.cc:654] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.816577 7122 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:46.818924 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 3697989873e94a6fad3835e0ebb36fce (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.819375 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3697989873e94a6fad3835e0ebb36fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.822948 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 3697989873e94a6fad3835e0ebb36fce (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:46.823438 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3697989873e94a6fad3835e0ebb36fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.824812 6958 ts_tablet_manager.cc:902] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:46.825743 7123 tablet_bootstrap.cc:492] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:46.830482 7123 tablet_bootstrap.cc:654] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.833168 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]), partition=
I20240501 14:00:46.833550 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.843699 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]), partition=
I20240501 14:00:46.843751 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]), partition=
I20240501 14:00:46.844087 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.844111 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.851884 7124 tablet_bootstrap.cc:492] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:46.853137 7122 tablet_bootstrap.cc:654] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.860203 7122 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:46.860428 7122 ts_tablet_manager.cc:1367] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.003s
I20240501 14:00:46.861326 7122 raft_consensus.cc:348] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.861580 7122 raft_consensus.cc:374] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.861680 7122 raft_consensus.cc:729] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.861922 7122 consensus_queue.cc:260] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.865779 7122 ts_tablet_manager.cc:1398] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:00:46.872126 7130 tablet_bootstrap.cc:492] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:46.873915 7130 tablet_bootstrap.cc:654] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.879640 7127 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:46.880522 2408 raft_consensus.cc:1203] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:46.882628 7127 tablet_bootstrap.cc:654] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.884541 7123 tablet_bootstrap.cc:492] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:46.885210 7123 ts_tablet_manager.cc:1367] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.000s
I20240501 14:00:46.886147 7129 tablet_bootstrap.cc:492] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:46.886230 7123 raft_consensus.cc:348] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.886611 7123 raft_consensus.cc:374] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.886680 7124 ts_tablet_manager.cc:1367] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.000s
I20240501 14:00:46.886757 7123 raft_consensus.cc:729] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.887073 7123 consensus_queue.cc:260] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.889786 7124 raft_consensus.cc:348] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.890848 7124 raft_consensus.cc:374] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.893314 7130 tablet_bootstrap.cc:492] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:46.893887 7124 raft_consensus.cc:729] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.894624 6958 tablet_copy_client.cc:308] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:00:46.895409 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 238aa15a2cd74ff69e17a2934b32b70d from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:38303)
I20240501 14:00:46.895630 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 238aa15a2cd74ff69e17a2934b32b70d from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:38303: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-238aa15a2cd74ff69e17a2934b32b70d
I20240501 14:00:46.895745 7129 tablet_bootstrap.cc:654] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.895020 7124 consensus_queue.cc:260] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.900074 2298 tablet_copy_source_session.cc:205] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:46.900223 7123 ts_tablet_manager.cc:1398] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:00:46.901314 7124 ts_tablet_manager.cc:1398] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:00:46.901854 7130 ts_tablet_manager.cc:1367] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20240501 14:00:46.903221 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238aa15a2cd74ff69e17a2934b32b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:46.906072 7124 tablet_bootstrap.cc:492] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:46.907441 7127 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:46.907938 7123 tablet_bootstrap.cc:492] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:46.910326 7127 ts_tablet_manager.cc:1367] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20240501 14:00:46.910986 7130 raft_consensus.cc:348] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.911303 7130 raft_consensus.cc:374] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.911425 7130 raft_consensus.cc:729] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.911667 7130 consensus_queue.cc:260] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.912242 7130 ts_tablet_manager.cc:1398] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:00:46.913568 7123 tablet_bootstrap.cc:654] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.913569 7127 raft_consensus.cc:348] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.913779 7127 raft_consensus.cc:374] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.913939 7127 raft_consensus.cc:729] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.914222 7127 consensus_queue.cc:260] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.918630 7127 ts_tablet_manager.cc:1398] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:00:46.919906 7124 tablet_bootstrap.cc:654] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.925107 7131 tablet_bootstrap.cc:492] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:46.928628 7131 tablet_bootstrap.cc:654] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.929414 6958 tablet_copy_client.cc:783] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:46.929687 6958 tablet_copy_client.cc:647] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:46.930090 7123 tablet_bootstrap.cc:492] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:46.930267 7123 ts_tablet_manager.cc:1367] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 14:00:46.931056 7123 raft_consensus.cc:348] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.931283 7123 raft_consensus.cc:374] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.931354 7123 raft_consensus.cc:729] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.931377 7129 tablet_bootstrap.cc:492] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:46.931535 7129 ts_tablet_manager.cc:1367] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.004s
I20240501 14:00:46.931736 6958 tablet_copy_client.cc:523] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:46.931596 7123 consensus_queue.cc:260] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.932197 7123 ts_tablet_manager.cc:1398] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:46.933228 7129 raft_consensus.cc:348] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.933594 7129 raft_consensus.cc:374] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.933696 7129 raft_consensus.cc:729] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.933923 7129 consensus_queue.cc:260] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.937687 7129 ts_tablet_manager.cc:1398] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:46.946327 6958 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:46.946938 7019 raft_consensus.cc:1050] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:46.947865 7019 consensus_queue.cc:237] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:46.956097 7124 tablet_bootstrap.cc:492] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:46.956395 7124 ts_tablet_manager.cc:1367] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240501 14:00:46.958971 7013 raft_consensus.cc:482] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.959129 7013 raft_consensus.cc:504] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.959810 7013 leader_election.cc:288] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:46.960008 2146 raft_consensus.cc:1203] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:46.961519 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dff7dc57d1e4b0493778a54bc2c2c5b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:46.962831 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dff7dc57d1e4b0493778a54bc2c2c5b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:46.963045 4900 raft_consensus.cc:2454] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
W20240501 14:00:46.963972 5230 leader_election.cc:341] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:46.964370 5229 leader_election.cc:302] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:46.967026 7013 raft_consensus.cc:2781] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.967157 7013 raft_consensus.cc:482] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.967264 7013 raft_consensus.cc:3037] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.970064 7013 raft_consensus.cc:504] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.971832 6686 raft_consensus.cc:482] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.972025 6686 raft_consensus.cc:504] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.972740 6686 leader_election.cc:288] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:46.974128 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52eb73b91ed441199eade33e66b84b39" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:46.974452 2408 raft_consensus.cc:1261] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:46.974443 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52eb73b91ed441199eade33e66b84b39" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:46.975135 2533 raft_consensus.cc:2454] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:46.975082 6878 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:46.975554 5097 leader_election.cc:302] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:46.976920 6686 raft_consensus.cc:2781] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.977017 6686 raft_consensus.cc:482] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.977108 6686 raft_consensus.cc:3037] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:46.977247 5099 leader_election.cc:341] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:46.980175 6686 raft_consensus.cc:504] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:46.980441 7051 raft_consensus.cc:482] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:46.980599 7051 raft_consensus.cc:504] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.981145 7013 leader_election.cc:288] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:46.981309 7051 leader_election.cc:288] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:46.981717 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dff7dc57d1e4b0493778a54bc2c2c5b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:46.982481 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697989873e94a6fad3835e0ebb36fce" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:46.982564 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697989873e94a6fad3835e0ebb36fce" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:46.982713 2406 raft_consensus.cc:2454] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:46.982659 7124 raft_consensus.cc:348] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:46.982859 7124 raft_consensus.cc:374] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.982923 7124 raft_consensus.cc:729] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.983153 7124 consensus_queue.cc:260] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
W20240501 14:00:46.983397 5230 leader_election.cc:341] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:46.983496 4834 leader_election.cc:302] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:46.983630 5293 raft_consensus.cc:2454] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:46.983750 7124 ts_tablet_manager.cc:1398] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240501 14:00:46.984061 7124 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:46.984265 7051 raft_consensus.cc:2781] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:46.984372 7051 raft_consensus.cc:482] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:46.984447 7051 raft_consensus.cc:3037] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.985502 5164 raft_consensus.cc:1261] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:46.985772 7124 tablet_bootstrap.cc:654] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:46.987233 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dff7dc57d1e4b0493778a54bc2c2c5b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:46.987452 4900 raft_consensus.cc:3037] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.988704 6878 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:00:46.991678 5965 raft_consensus.cc:1050] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:46.992295 7131 tablet_bootstrap.cc:492] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:46.992453 7131 ts_tablet_manager.cc:1367] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.003s
I20240501 14:00:46.992777 5965 consensus_queue.cc:237] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:46.993373 7131 raft_consensus.cc:348] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:46.993567 7131 raft_consensus.cc:374] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:46.993652 7131 raft_consensus.cc:729] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:46.995630 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52eb73b91ed441199eade33e66b84b39" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:46.995880 2538 raft_consensus.cc:3037] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:46.999123 7131 consensus_queue.cc:260] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:47.001152 2533 raft_consensus.cc:1261] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.004319 7131 ts_tablet_manager.cc:1398] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240501 14:00:47.006878 2538 raft_consensus.cc:2454] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:47.007539 5097 leader_election.cc:302] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:47.007886 7049 raft_consensus.cc:2781] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.008025 7049 raft_consensus.cc:686] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:47.008335 7049 consensus_queue.cc:237] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.010054 6878 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.012220 7131 tablet_bootstrap.cc:492] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:46.987095 7051 raft_consensus.cc:504] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:47.013541 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697989873e94a6fad3835e0ebb36fce" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:47.013821 5293 raft_consensus.cc:3037] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.016110 7131 tablet_bootstrap.cc:654] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.017021 5293 raft_consensus.cc:2454] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:47.017554 4836 leader_election.cc:302] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:47.017889 6948 raft_consensus.cc:2781] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.018054 6948 raft_consensus.cc:686] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:47.018343 6948 consensus_queue.cc:237] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:47.023207 5034 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.023877 6821 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.025602 2146 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.026403 7158 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.027572 5293 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.029129 4900 raft_consensus.cc:2454] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:47.029770 5229 leader_election.cc:302] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:47.030258 7013 raft_consensus.cc:2781] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.030397 7013 raft_consensus.cc:686] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:47.030827 7013 consensus_queue.cc:237] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:47.031911 7159 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.034633 7052 raft_consensus.cc:1050] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:00:47.036041 7052 consensus_queue.cc:237] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:47.037041 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697989873e94a6fad3835e0ebb36fce" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:47.037288 2408 raft_consensus.cc:3037] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.040256 2408 raft_consensus.cc:2454] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:47.043936 7051 leader_election.cc:288] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:47.051290 6764 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.055955 5164 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.058944 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52eb73b91ed441199eade33e66b84b39" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
W20240501 14:00:47.059538 5099 leader_election.cc:341] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.059899 6686 leader_election.cc:288] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:47.060134 2406 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.064155 2275 raft_consensus.cc:1261] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.064465 2000 catalog_manager.cc:5484] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.064559 5965 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.066434 2533 raft_consensus.cc:1261] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.070115 5293 raft_consensus.cc:1261] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.076083 5293 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.076570 2275 consensus_queue.cc:237] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:47.078902 5164 raft_consensus.cc:1261] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.079564 6764 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.081140 2147 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.083628 5164 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.086064 6545 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20240501 14:00:47.088230 6878 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.090621 2000 catalog_manager.cc:5484] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.108254 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73940622ad734b88af31e9fa0c1000e5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.116329 6545 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20240501 14:00:47.117990 7052 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240501 14:00:47.117498 7164 log.cc:924] Time spent T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240501 14:00:47.143157 1999 catalog_manager.cc:5484] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.148051 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 73940622ad734b88af31e9fa0c1000e5 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
W20240501 14:00:47.167200 2343 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.8:34221, user_credentials={real_user=slave}} blocked reactor thread for 96381.4us
I20240501 14:00:47.167984 6958 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:47.168470 6958 tablet_bootstrap.cc:492] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:00:47.168787 6958 ts_tablet_manager.cc:1367] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.223s user 0.084s sys 0.053s
I20240501 14:00:47.169589 6958 raft_consensus.cc:348] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:00:47.169867 6958 raft_consensus.cc:729] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:00:47.170073 6958 consensus_queue.cc:260] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
W20240501 14:00:47.170238 4836 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.6:46037, user_credentials={real_user=slave}} blocked reactor thread for 151795us
I20240501 14:00:47.173213 6958 ts_tablet_manager.cc:1398] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:47.173962 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-238aa15a2cd74ff69e17a2934b32b70d received from {username='slave'} at 127.28.17.7:38303
I20240501 14:00:47.174085 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-238aa15a2cd74ff69e17a2934b32b70d on tablet 238aa15a2cd74ff69e17a2934b32b70d with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:47.176936 2408 raft_consensus.cc:1203] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:47.177191 2406 raft_consensus.cc:1261] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.177860 7019 consensus_queue.cc:1035] T 73940622ad734b88af31e9fa0c1000e5 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.184219 2406 raft_consensus.cc:2932] T 73940622ad734b88af31e9fa0c1000e5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.193125 2000 catalog_manager.cc:5484] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.195639 7104 tablet_replica.cc:330] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:47.195915 7104 raft_consensus.cc:2229] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.196221 7104 pending_rounds.cc:62] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Trying to abort 1 pending ops.
I20240501 14:00:47.196355 7104 pending_rounds.cc:68] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022887104295968768 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "73940622ad734b88af31e9fa0c1000e5" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } } }
I20240501 14:00:47.196806 7104 raft_consensus.cc:2894] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:00:47.196933 7104 raft_consensus.cc:2866] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:00:47.197027 7104 raft_consensus.cc:2258] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.193234 1999 catalog_manager.cc:5484] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.207652 7104 ts_tablet_manager.cc:1875] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.216452 7052 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:47.219723 2275 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:47.233649 2533 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:47.236974 5164 raft_consensus.cc:1203] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:47.238093 2001 catalog_manager.cc:5484] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:47.251192 4836 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.28.17.62:33799, user_credentials={real_user=slave}} blocked reactor thread for 73299.7us
I20240501 14:00:47.252087 7100 raft_consensus.cc:1050] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:47.252902 7100 consensus_queue.cc:237] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:47.259794 7131 tablet_bootstrap.cc:492] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:47.262176 2006 catalog_manager.cc:5484] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.264439 7131 ts_tablet_manager.cc:1367] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.252s user 0.037s sys 0.073s
I20240501 14:00:47.267904 5295 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:47.265271 7131 raft_consensus.cc:348] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.270888 7131 raft_consensus.cc:374] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.270985 7131 raft_consensus.cc:729] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.271049 4901 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.271242 7131 consensus_queue.cc:260] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.271673 2406 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.271834 7131 ts_tablet_manager.cc:1398] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 14:00:47.272197 7100 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.273129 7124 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:47.273288 7124 ts_tablet_manager.cc:1367] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.289s user 0.046s sys 0.149s
I20240501 14:00:47.273697 2147 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.274014 7124 raft_consensus.cc:348] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.274217 7124 raft_consensus.cc:374] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.274307 7124 raft_consensus.cc:729] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.275087 7100 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.276777 2408 consensus_queue.cc:237] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:00:47.277166 6878 raft_consensus.cc:482] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.277346 6878 raft_consensus.cc:504] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.278357 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e2cdb9f91e4ab9a4a5d8db0189e16f" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:47.278313 7124 consensus_queue.cc:260] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.278659 5034 raft_consensus.cc:2454] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:47.278913 7124 ts_tablet_manager.cc:1398] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:47.279072 2213 leader_election.cc:302] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:47.281569 7124 tablet_bootstrap.cc:492] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:47.283754 7124 tablet_bootstrap.cc:654] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.284893 5296 raft_consensus.cc:1261] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.282482 6764 raft_consensus.cc:2781] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.285857 6764 raft_consensus.cc:482] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.285975 6764 raft_consensus.cc:3037] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.286228 7052 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.289003 6764 raft_consensus.cc:504] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.292429 6764 leader_election.cc:288] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:47.292353 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e2cdb9f91e4ab9a4a5d8db0189e16f" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
W20240501 14:00:47.293064 2213 leader_election.cc:341] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.293227 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e2cdb9f91e4ab9a4a5d8db0189e16f" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:47.292889 5034 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.293437 5032 raft_consensus.cc:3037] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.293498 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e2cdb9f91e4ab9a4a5d8db0189e16f" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
W20240501 14:00:47.293962 2213 leader_election.cc:341] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.296584 5032 raft_consensus.cc:2454] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:47.296592 5164 consensus_queue.cc:237] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:47.298024 2213 leader_election.cc:302] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:47.291836 6878 leader_election.cc:288] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:47.301406 6878 raft_consensus.cc:2781] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.301690 6878 raft_consensus.cc:686] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:47.302109 6878 consensus_queue.cc:237] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.304271 2533 raft_consensus.cc:1261] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.304962 7052 consensus_queue.cc:1035] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.305732 5296 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.306366 7100 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.311827 7158 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.315886 6905 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.316227 6545 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:47.322022 2406 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.324826 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dcb6951344ad49b3aee27205e898c347 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.328303 1999 catalog_manager.cc:5484] T dcb6951344ad49b3aee27205e898c347 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.329042 5296 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:47.331241 2538 raft_consensus.cc:2932] T dcb6951344ad49b3aee27205e898c347 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:00:47.333420 7104 ts_tablet_manager.cc:1888] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:47.335841 4901 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.341588 7158 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.346311 5032 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.347136 7159 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.349140 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44a4e92b144f47b5800506f4bc3eadb6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.349612 1997 catalog_manager.cc:5484] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.349628 5296 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.350379 7104 log.cc:1196] T 73940622ad734b88af31e9fa0c1000e5 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/73940622ad734b88af31e9fa0c1000e5
I20240501 14:00:47.351920 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 73940622ad734b88af31e9fa0c1000e5 (table rebalance_test_table_5 [id=eb01625d83a14fe5bdd8ce6f8834012b]) successfully deleted
I20240501 14:00:47.354530 2001 catalog_manager.cc:5484] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.355749 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 44a4e92b144f47b5800506f4bc3eadb6 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:47.363831 1999 catalog_manager.cc:5484] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.364567 5032 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:47.373632 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb6951344ad49b3aee27205e898c347 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:47.377115 7019 raft_consensus.cc:1050] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:47.378260 7019 consensus_queue.cc:237] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:47.383731 7124 tablet_bootstrap.cc:492] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:47.386476 7124 ts_tablet_manager.cc:1367] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.107s user 0.016s sys 0.015s
I20240501 14:00:47.387929 7124 raft_consensus.cc:348] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.386122 2146 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.389027 6653 raft_consensus.cc:482] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.389266 6653 raft_consensus.cc:504] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:47.390370 6653 leader_election.cc:288] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:47.391602 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dff7dc57d1e4b0493778a54bc2c2c5b" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:47.392004 4901 raft_consensus.cc:2379] T 3dff7dc57d1e4b0493778a54bc2c2c5b P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in current term 1: Already voted for candidate 5a4b9a96e9734baa8a03d6056bfa934c in this term.
I20240501 14:00:47.393873 5164 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.395059 2538 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.395393 7191 tablet_replica.cc:330] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:47.395583 7191 raft_consensus.cc:2229] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.396061 7191 raft_consensus.cc:2258] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.397049 6878 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.397122 7124 raft_consensus.cc:374] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.397229 7124 raft_consensus.cc:729] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.397444 7181 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.392962 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dff7dc57d1e4b0493778a54bc2c2c5b" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:47.398394 7124 consensus_queue.cc:260] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.398648 4969 leader_election.cc:302] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:47.399111 6653 raft_consensus.cc:3037] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.399127 2147 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.400586 6878 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.401760 6653 raft_consensus.cc:2726] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:47.401973 2537 consensus_queue.cc:237] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:47.407058 7124 ts_tablet_manager.cc:1398] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:00:47.412205 4901 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.415500 7100 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.417685 7191 ts_tablet_manager.cc:1875] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.418643 7100 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.419502 4901 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.423816 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1cb2dab6f4684bb18bddc9edba1d6517 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.428246 7019 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.427335 1999 catalog_manager.cc:5484] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.428920 2538 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.432127 6653 raft_consensus.cc:1050] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:00:47.432996 6653 consensus_queue.cc:237] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:47.433389 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb2dab6f4684bb18bddc9edba1d6517 with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:47.434068 7196 tablet_replica.cc:330] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:47.434350 7196 raft_consensus.cc:2229] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.434903 7196 raft_consensus.cc:2258] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.437973 2000 catalog_manager.cc:5484] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.442709 2143 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.446619 2147 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.447629 7013 consensus_queue.cc:1035] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.447932 6905 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.449306 2147 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:47.450562 7196 ts_tablet_manager.cc:1875] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.467075 7188 tablet_replica.cc:330] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:47.467092 7196 ts_tablet_manager.cc:1888] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:47.467234 7196 log.cc:1196] T 1cb2dab6f4684bb18bddc9edba1d6517 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/1cb2dab6f4684bb18bddc9edba1d6517
I20240501 14:00:47.468189 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 1cb2dab6f4684bb18bddc9edba1d6517 (table rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]) successfully deleted
I20240501 14:00:47.475003 7188 raft_consensus.cc:2229] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.475629 7049 raft_consensus.cc:1050] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER d1a414d381f44829a2510cb76d481887 to VOTER
I20240501 14:00:47.476285 7049 consensus_queue.cc:237] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:00:47.476589 7188 raft_consensus.cc:2258] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.478019 2146 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.478636 4901 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.478753 6929 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.479710 6929 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.483526 7188 ts_tablet_manager.cc:1875] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.492012 5164 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.494695 7191 ts_tablet_manager.cc:1888] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:47.494863 7191 log.cc:1196] T dcb6951344ad49b3aee27205e898c347 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/dcb6951344ad49b3aee27205e898c347
I20240501 14:00:47.495383 2275 consensus_queue.cc:237] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:47.498793 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet dcb6951344ad49b3aee27205e898c347 (table rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]) successfully deleted
I20240501 14:00:47.499372 2146 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.500051 5296 raft_consensus.cc:3037] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.500043 7019 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.500449 2406 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.502452 7019 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.505074 2144 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.506546 7188 ts_tablet_manager.cc:1888] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:47.506680 7188 log.cc:1196] T 44a4e92b144f47b5800506f4bc3eadb6 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/44a4e92b144f47b5800506f4bc3eadb6
I20240501 14:00:47.507088 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 65306dbef405443fb4c7188ccb762b83 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.509352 7155 consensus_queue.cc:1035] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.509974 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:47.514110 2001 catalog_manager.cc:5484] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.516582 6653 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.517593 2274 raft_consensus.cc:1261] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.518257 7159 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.519404 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 44a4e92b144f47b5800506f4bc3eadb6 (table rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]) successfully deleted
I20240501 14:00:47.520967 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 65306dbef405443fb4c7188ccb762b83 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:47.521297 7104 tablet_replica.cc:330] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:47.521508 7104 raft_consensus.cc:2229] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.521710 7104 raft_consensus.cc:2258] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.522933 7104 ts_tablet_manager.cc:1875] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.524865 2143 raft_consensus.cc:1261] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.525692 7158 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.528301 5295 raft_consensus.cc:1261] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.529614 7104 ts_tablet_manager.cc:1888] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:47.529836 7104 log.cc:1196] T 65306dbef405443fb4c7188ccb762b83 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/65306dbef405443fb4c7188ccb762b83
I20240501 14:00:47.530725 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 65306dbef405443fb4c7188ccb762b83 (table rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]) successfully deleted
I20240501 14:00:47.533207 7159 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.534361 7159 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.535599 5295 raft_consensus.cc:3037] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.536080 5034 raft_consensus.cc:1261] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:47.538856 7181 consensus_queue.cc:1035] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.539731 5295 raft_consensus.cc:1261] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:47.540382 7181 consensus_queue.cc:1035] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.547230 7155 consensus_queue.cc:1035] T 3697989873e94a6fad3835e0ebb36fce P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.569442 7159 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.585475 6881 consensus_queue.cc:1035] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.587909 6929 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.592950 6821 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.598161 2408 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.605020 4900 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.610627 2001 catalog_manager.cc:5484] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.616896 2274 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.611887 2144 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.622834 5296 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.629370 6947 raft_consensus.cc:1050] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:00:47.630580 6947 consensus_queue.cc:237] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:47.633348 2000 catalog_manager.cc:5484] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.639796 2143 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.642666 5161 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:47.643457 6878 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:47.645457 5164 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.649238 5161 consensus_queue.cc:237] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:00:47.657435 5296 raft_consensus.cc:1261] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.659263 2001 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]). Expected version 1 got 0
I20240501 14:00:47.663226 2143 raft_consensus.cc:1261] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.674193 2408 raft_consensus.cc:1261] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.677377 7155 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240501 14:00:47.678422 2001 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]). Expected version 1 got 0
I20240501 14:00:47.680212 7226 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:47.681910 5034 consensus_queue.cc:237] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:47.682199 2144 raft_consensus.cc:1261] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.683063 6947 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.686403 2406 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.686769 7226 tablet.cc:1620] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:47.687256 6929 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.689277 2147 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.689905 6929 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.691205 6653 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.695484 2408 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.676200 7228 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:47.699965 7228 tablet.cc:1620] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:47.700368 2001 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]). Expected version 1 got 0
I20240501 14:00:47.704191 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5086890bd254edebdf12812ff5c3728 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.708793 1999 catalog_manager.cc:5484] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.710659 7159 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20240501 14:00:47.711407 2144 raft_consensus.cc:2932] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.712862 7157 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.715643 7224 consensus_queue.cc:1035] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240501 14:00:47.716683 5164 raft_consensus.cc:1261] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:47.720319 2147 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.720618 7224 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.721704 2006 catalog_manager.cc:5484] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.723176 2146 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.726465 2408 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.728384 6878 raft_consensus.cc:1050] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:00:47.729579 6878 consensus_queue.cc:237] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:47.742122 1997 catalog_manager.cc:5484] T 13d8ec926b7a46568470e824763569e6 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.743331 2001 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23] alter complete (version 1)
I20240501 14:00:47.749713 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d8ec926b7a46568470e824763569e6 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:47.750123 7191 tablet_replica.cc:330] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:47.750339 7191 raft_consensus.cc:2229] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.750824 7191 raft_consensus.cc:2258] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.751823 7191 ts_tablet_manager.cc:1875] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.752655 6881 raft_consensus.cc:482] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:00:47.752869 6881 raft_consensus.cc:504] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.753953 6881 leader_election.cc:288] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:47.755473 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e2cdb9f91e4ab9a4a5d8db0189e16f" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:47.756208 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e2cdb9f91e4ab9a4a5d8db0189e16f" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:47.758960 5230 leader_election.cc:302] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:47.760660 6881 raft_consensus.cc:2726] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:00:47.762415 2001 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]). Expected version 1 got 0
I20240501 14:00:47.779151 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet d5086890bd254edebdf12812ff5c3728 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:47.785434 2000 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23]). Expected version 1 got 0
I20240501 14:00:47.781849 7240 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:47.788604 7240 tablet.cc:1620] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:47.789371 6947 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:47.789971 7241 tablet_replica.cc:330] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:47.790176 7241 raft_consensus.cc:2229] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:47.790422 7241 raft_consensus.cc:2258] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:47.790757 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 13d8ec926b7a46568470e824763569e6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.792783 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:47.797866 4900 consensus_queue.cc:237] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:00:47.800300 7191 ts_tablet_manager.cc:1888] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:47.800451 7191 log.cc:1196] T 13d8ec926b7a46568470e824763569e6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/13d8ec926b7a46568470e824763569e6
I20240501 14:00:47.802870 2146 raft_consensus.cc:1261] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.803373 2408 raft_consensus.cc:1261] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:47.804095 7157 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.806839 6947 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.807699 7157 consensus_queue.cc:1035] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:47.808018 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 13d8ec926b7a46568470e824763569e6 (table rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]) successfully deleted
I20240501 14:00:47.809046 2146 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.810926 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aaaae937ea4d44068d3ad5b9024b9421 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:47.819134 2408 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.820292 5014 tablet_service.cc:1467] Processing CreateTablet for tablet f30705f9c298491dbfb32bdc554647e4 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.820730 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30705f9c298491dbfb32bdc554647e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.822790 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 0724625a6a8e4ce38e8087472b856c5c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.823983 2125 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4e5590d124918ac4814a86b006b69 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.824239 5274 tablet_service.cc:1467] Processing CreateTablet for tablet f30705f9c298491dbfb32bdc554647e4 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.824407 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4e5590d124918ac4814a86b006b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.824616 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30705f9c298491dbfb32bdc554647e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.825424 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f30705f9c298491dbfb32bdc554647e4 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.825819 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30705f9c298491dbfb32bdc554647e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.827556 2538 raft_consensus.cc:1261] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.827822 2143 raft_consensus.cc:1261] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.828862 5164 raft_consensus.cc:1261] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:47.829619 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 0724625a6a8e4ce38e8087472b856c5c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.829735 4881 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4e5590d124918ac4814a86b006b69 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.830030 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0724625a6a8e4ce38e8087472b856c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.830159 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4e5590d124918ac4814a86b006b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.832195 6764 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.832753 6764 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.833204 6764 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:47.841251 1997 catalog_manager.cc:5484] T aaaae937ea4d44068d3ad5b9024b9421 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.852432 7246 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:47.853278 5275 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4e5590d124918ac4814a86b006b69 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.853783 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4e5590d124918ac4814a86b006b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.856539 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 0724625a6a8e4ce38e8087472b856c5c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:47.857949 7181 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.853883 5293 raft_consensus.cc:2932] T 13d8ec926b7a46568470e824763569e6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:00:47.860383 7246 tablet_bootstrap.cc:654] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.872452 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0724625a6a8e4ce38e8087472b856c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.880040 5161 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.887689 2001 catalog_manager.cc:5484] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.890126 2004 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:47.895381 5164 raft_consensus.cc:2932] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:00:47.900408 2538 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.915848 7243 tablet_bootstrap.cc:492] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:47.917076 7226 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:47.917507 7243 tablet_bootstrap.cc:654] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.917577 7226 tablet.cc:1620] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:47.918027 7228 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:47.919085 7249 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:47.919162 7228 tablet.cc:1620] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:47.920805 7249 tablet_bootstrap.cc:654] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.922960 7241 ts_tablet_manager.cc:1875] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:47.927313 7240 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:47.927865 7240 tablet.cc:1620] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:47.933857 2143 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:47.940992 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0724625a6a8e4ce38e8087472b856c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.948763 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet aaaae937ea4d44068d3ad5b9024b9421 with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:47.964519 7251 tablet_bootstrap.cc:492] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:47.969341 7243 tablet_bootstrap.cc:492] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:47.969614 7243 ts_tablet_manager.cc:1367] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.006s
I20240501 14:00:47.970597 7243 raft_consensus.cc:348] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.970942 7243 raft_consensus.cc:374] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.971060 7243 raft_consensus.cc:729] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.971333 7243 consensus_queue.cc:260] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:47.972373 7243 ts_tablet_manager.cc:1398] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:47.973357 7254 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:47.978049 7241 ts_tablet_manager.cc:1888] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:47.979387 7254 tablet_bootstrap.cc:654] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.967653 7251 tablet_bootstrap.cc:654] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.982748 7243 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:47.987375 7241 log.cc:1196] T d5086890bd254edebdf12812ff5c3728 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/d5086890bd254edebdf12812ff5c3728
I20240501 14:00:47.981768 7249 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:47.988389 7249 ts_tablet_manager.cc:1367] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.000s
I20240501 14:00:47.989346 7253 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:47.989375 7249 raft_consensus.cc:348] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:47.989760 7249 raft_consensus.cc:374] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.989899 7249 raft_consensus.cc:729] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.990320 7249 consensus_queue.cc:260] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:47.992549 7249 ts_tablet_manager.cc:1398] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:47.994231 7243 tablet_bootstrap.cc:654] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.995445 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet d5086890bd254edebdf12812ff5c3728 (table rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]) successfully deleted
I20240501 14:00:48.002103 7251 tablet_bootstrap.cc:492] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:48.002317 7251 ts_tablet_manager.cc:1367] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20240501 14:00:48.003717 7251 raft_consensus.cc:348] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:48.003934 7251 raft_consensus.cc:374] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.004017 7251 raft_consensus.cc:729] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.004241 7251 consensus_queue.cc:260] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:48.009244 7256 tablet_replica.cc:330] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:48.001578 2274 consensus_queue.cc:237] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:00:48.016522 7254 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:48.016712 7254 ts_tablet_manager.cc:1367] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20240501 14:00:48.017614 7254 raft_consensus.cc:348] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.017860 7254 raft_consensus.cc:374] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.017940 7254 raft_consensus.cc:729] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.018199 7254 consensus_queue.cc:260] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.018807 7254 ts_tablet_manager.cc:1398] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:48.019131 7254 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:48.020992 7254 tablet_bootstrap.cc:654] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.004792 7251 ts_tablet_manager.cc:1398] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:48.037154 2143 raft_consensus.cc:1261] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:48.037902 6878 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:48.042021 6878 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:48.043434 7254 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:48.043593 7254 ts_tablet_manager.cc:1367] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240501 14:00:48.044334 7254 raft_consensus.cc:348] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.044605 7254 raft_consensus.cc:374] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.044669 7254 raft_consensus.cc:729] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.044916 7254 consensus_queue.cc:260] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.046133 7254 ts_tablet_manager.cc:1398] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:48.046474 7254 tablet_bootstrap.cc:492] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:48.048103 7254 tablet_bootstrap.cc:654] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.048058 2146 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:48.053109 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 238aa15a2cd74ff69e17a2934b32b70d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:48.059500 1999 catalog_manager.cc:5484] T 238aa15a2cd74ff69e17a2934b32b70d P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:48.063891 2001 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=2dd881d821444981964200c757dc7d23] alter complete (version 2)
I20240501 14:00:48.069411 7254 tablet_bootstrap.cc:492] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:48.069617 7254 ts_tablet_manager.cc:1367] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.006s
I20240501 14:00:48.070487 7254 raft_consensus.cc:348] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:48.070757 7254 raft_consensus.cc:374] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.070850 7254 raft_consensus.cc:729] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.071097 7254 consensus_queue.cc:260] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:48.071700 7254 ts_tablet_manager.cc:1398] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:48.072743 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 238aa15a2cd74ff69e17a2934b32b70d with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:48.080102 7243 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:48.080379 7243 ts_tablet_manager.cc:1367] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.004s
I20240501 14:00:48.081315 7243 raft_consensus.cc:348] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.081599 7243 raft_consensus.cc:374] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.081694 7243 raft_consensus.cc:729] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.081959 7243 consensus_queue.cc:260] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.082624 7243 ts_tablet_manager.cc:1398] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:48.092330 7256 raft_consensus.cc:2229] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.020226 5164 raft_consensus.cc:1261] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:48.095541 7256 raft_consensus.cc:2258] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.095734 6878 consensus_queue.cc:1035] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:48.096889 7256 ts_tablet_manager.cc:1875] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:48.098255 7253 tablet_bootstrap.cc:654] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.097465 5164 raft_consensus.cc:2932] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:00:48.100751 7268 tablet_replica.cc:330] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:48.104219 7268 raft_consensus.cc:2229] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.104517 7268 raft_consensus.cc:2258] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.106491 7268 ts_tablet_manager.cc:1875] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:48.108922 7246 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:48.109431 7246 ts_tablet_manager.cc:1367] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.257s user 0.029s sys 0.040s
I20240501 14:00:48.110392 7246 raft_consensus.cc:348] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.110775 7246 raft_consensus.cc:374] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.110967 7246 raft_consensus.cc:729] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.111378 7246 consensus_queue.cc:260] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.113282 7246 ts_tablet_manager.cc:1398] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:48.113837 7256 ts_tablet_manager.cc:1888] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:48.114023 7256 log.cc:1196] T aaaae937ea4d44068d3ad5b9024b9421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/aaaae937ea4d44068d3ad5b9024b9421
I20240501 14:00:48.114547 7253 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:48.114780 7253 ts_tablet_manager.cc:1367] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.001s
I20240501 14:00:48.115307 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet aaaae937ea4d44068d3ad5b9024b9421 (table rebalancer_extra_table_0 [id=860283adb5c845fd81ea13cbb1157ee4]) successfully deleted
I20240501 14:00:48.119519 7253 raft_consensus.cc:348] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.119758 7253 raft_consensus.cc:374] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.119854 7253 raft_consensus.cc:729] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.120081 7253 consensus_queue.cc:260] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.120635 7253 ts_tablet_manager.cc:1398] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:00:48.125535 7268 ts_tablet_manager.cc:1888] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:48.125664 7268 log.cc:1196] T 238aa15a2cd74ff69e17a2934b32b70d P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/238aa15a2cd74ff69e17a2934b32b70d
I20240501 14:00:48.126432 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 238aa15a2cd74ff69e17a2934b32b70d (table rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]) successfully deleted
I20240501 14:00:48.157402 2408 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:48.158566 6653 raft_consensus.cc:482] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.158725 6653 raft_consensus.cc:504] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:48.159433 6653 leader_election.cc:288] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:48.162716 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0724625a6a8e4ce38e8087472b856c5c" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:48.162959 2406 raft_consensus.cc:2454] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:48.163746 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0724625a6a8e4ce38e8087472b856c5c" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:48.163959 5293 raft_consensus.cc:2454] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:48.164877 4967 leader_election.cc:302] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:48.167927 6653 raft_consensus.cc:2781] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.168061 6653 raft_consensus.cc:482] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.168151 6653 raft_consensus.cc:3037] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:48.172964 2342 consensus_peers.cc:485] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd3f8661a13944098a18d3eea7ddeb00. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:48.174052 2143 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:48.174299 6653 raft_consensus.cc:504] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:48.174417 2274 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:48.175594 7166 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:48.176182 7166 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:48.183261 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0724625a6a8e4ce38e8087472b856c5c" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:48.183569 2406 raft_consensus.cc:3037] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.185556 6927 raft_consensus.cc:482] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.185716 6927 raft_consensus.cc:504] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.186405 6927 leader_election.cc:288] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:48.186776 6878 raft_consensus.cc:482] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.186960 6878 raft_consensus.cc:504] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.187276 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:48.187392 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:48.187544 5292 raft_consensus.cc:2454] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:48.187613 2275 raft_consensus.cc:2454] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:48.187652 6878 leader_election.cc:288] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:48.188813 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:48.188966 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:48.189030 5034 raft_consensus.cc:2454] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:48.189167 5292 raft_consensus.cc:2454] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:48.189368 2406 raft_consensus.cc:2454] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:48.189473 4967 leader_election.cc:302] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:48.189648 2213 leader_election.cc:302] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:48.189795 4967 leader_election.cc:302] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:48.189901 6653 leader_election.cc:288] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:48.190057 6653 raft_consensus.cc:2781] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.190152 6653 raft_consensus.cc:482] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.190178 6929 raft_consensus.cc:2781] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.190263 6653 raft_consensus.cc:3037] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.190306 6929 raft_consensus.cc:686] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:48.190615 6929 consensus_queue.cc:237] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:48.192229 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:48.192637 6878 raft_consensus.cc:2781] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.192749 6878 raft_consensus.cc:482] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.192844 6878 raft_consensus.cc:3037] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.195542 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0724625a6a8e4ce38e8087472b856c5c" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:48.195780 5292 raft_consensus.cc:3037] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.200570 6878 raft_consensus.cc:504] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.203197 6878 leader_election.cc:288] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:48.203459 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:48.204005 5292 raft_consensus.cc:2454] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:48.204643 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:48.204874 5293 raft_consensus.cc:3037] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.209400 5293 raft_consensus.cc:2454] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:48.209954 2213 leader_election.cc:302] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:48.216104 6878 raft_consensus.cc:2781] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.216635 6878 raft_consensus.cc:686] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:48.216924 6878 consensus_queue.cc:237] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.217792 7279 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:48.222190 6947 raft_consensus.cc:482] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.222417 6947 raft_consensus.cc:504] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.223145 6947 leader_election.cc:288] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:48.225026 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4e5590d124918ac4814a86b006b69" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:48.225443 5293 raft_consensus.cc:2454] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:48.226243 4836 leader_election.cc:302] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:48.226641 6947 raft_consensus.cc:2781] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.226737 6947 raft_consensus.cc:482] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.226874 6947 raft_consensus.cc:3037] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.228263 7226 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:48.229473 6947 raft_consensus.cc:504] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.230773 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4e5590d124918ac4814a86b006b69" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:48.230995 2146 raft_consensus.cc:3037] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.233899 2146 raft_consensus.cc:2454] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:48.234076 7240 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:48.230165 6947 leader_election.cc:288] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:48.234550 4834 leader_election.cc:302] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:48.235175 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4e5590d124918ac4814a86b006b69" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:48.235579 5293 raft_consensus.cc:3037] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.236575 6947 raft_consensus.cc:2781] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.236816 6947 raft_consensus.cc:686] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:48.237154 6947 consensus_queue.cc:237] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.239004 1999 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:48.243408 7228 tablet.cc:1722] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:48.245159 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4e5590d124918ac4814a86b006b69" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:48.245442 2143 raft_consensus.cc:2362] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1. Re-sending same reply.
I20240501 14:00:48.249114 2005 catalog_manager.cc:5484] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:48.251209 2143 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:48.252070 2274 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:00:48.254384 6653 raft_consensus.cc:504] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:48.254891 5032 raft_consensus.cc:2379] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in current term 1: Already voted for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in this term.
I20240501 14:00:48.258572 5293 raft_consensus.cc:2454] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:48.260564 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:48.260805 6653 leader_election.cc:288] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:48.267369 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:48.267670 5292 raft_consensus.cc:2379] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in current term 1: Already voted for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in this term.
I20240501 14:00:48.269398 1997 catalog_manager.cc:5484] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:48.270836 4969 leader_election.cc:302] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:48.277269 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=2dd881d821444981964200c757dc7d23] alter complete (version 3)
I20240501 14:00:48.284883 6929 raft_consensus.cc:2726] T f30705f9c298491dbfb32bdc554647e4 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:00:48.287050 1997 catalog_manager.cc:5484] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:48.306038 2408 consensus_queue.cc:237] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:48.308910 2143 raft_consensus.cc:1261] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:48.308940 2537 raft_consensus.cc:1261] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:48.309504 2343 consensus_peers.cc:485] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 457c62e1ddf547ad900ed529b3244474. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:48.309702 7279 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:48.310067 7169 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:48.320194 7169 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:48.321060 2146 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:48.321545 2537 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:48.327123 1997 catalog_manager.cc:5484] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:48.384425 1997 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:48.384739 1997 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:48.388443 1997 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f
I20240501 14:00:48.389421 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:48 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:48.389570 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:48 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:48.390056 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:48 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:48.406872 7302 tablet_replica.cc:330] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:48.406901 7303 tablet_replica.cc:330] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:48.407161 7303 raft_consensus.cc:2229] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.407364 7302 raft_consensus.cc:2229] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:48.407403 7303 raft_consensus.cc:2258] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.407895 7302 raft_consensus.cc:2258] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.408321 7303 ts_tablet_manager.cc:1875] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.408919 7302 ts_tablet_manager.cc:1875] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.411288 7304 tablet_replica.cc:330] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:48.413434 7304 raft_consensus.cc:2229] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:48.413772 7304 raft_consensus.cc:2258] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:48.414570 7303 ts_tablet_manager.cc:1888] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:48.414762 7303 log.cc:1196] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/74e2cdb9f91e4ab9a4a5d8db0189e16f
I20240501 14:00:48.414856 7304 ts_tablet_manager.cc:1875] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:48.415290 7303 ts_tablet_manager.cc:1909] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:48.416462 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table_new_name [id=2dd881d821444981964200c757dc7d23]) successfully deleted
I20240501 14:00:48.417156 7302 ts_tablet_manager.cc:1888] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:48.417593 7302 log.cc:1196] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/74e2cdb9f91e4ab9a4a5d8db0189e16f
I20240501 14:00:48.418083 7302 ts_tablet_manager.cc:1909] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:48.419173 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table_new_name [id=2dd881d821444981964200c757dc7d23]) successfully deleted
I20240501 14:00:48.431387 7304 ts_tablet_manager.cc:1888] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:48.431650 7304 log.cc:1196] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/74e2cdb9f91e4ab9a4a5d8db0189e16f
I20240501 14:00:48.432629 7304 ts_tablet_manager.cc:1909] T 74e2cdb9f91e4ab9a4a5d8db0189e16f P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:48.435740 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 74e2cdb9f91e4ab9a4a5d8db0189e16f (table rebalancer_dynamic_table_new_name [id=2dd881d821444981964200c757dc7d23]) successfully deleted
I20240501 14:00:48.491883 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:48.502661 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6ec775542d46ee901be28099092a63 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]), partition=
I20240501 14:00:48.503069 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6ec775542d46ee901be28099092a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.506356 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6ec775542d46ee901be28099092a63 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]), partition=
I20240501 14:00:48.506867 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6ec775542d46ee901be28099092a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.506907 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6ec775542d46ee901be28099092a63 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]), partition=
I20240501 14:00:48.507293 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6ec775542d46ee901be28099092a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.523423 7321 tablet_bootstrap.cc:492] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:48.523602 7254 tablet_bootstrap.cc:492] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:48.525437 7321 tablet_bootstrap.cc:654] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.525559 7254 tablet_bootstrap.cc:654] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.531579 7254 tablet_bootstrap.cc:492] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:48.531906 7321 tablet_bootstrap.cc:492] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:48.531903 7254 ts_tablet_manager.cc:1367] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240501 14:00:48.532150 7321 ts_tablet_manager.cc:1367] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240501 14:00:48.532743 7254 raft_consensus.cc:348] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.532912 7321 raft_consensus.cc:348] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.533116 7254 raft_consensus.cc:374] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.533183 7321 raft_consensus.cc:374] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.533237 7254 raft_consensus.cc:729] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.533320 7321 raft_consensus.cc:729] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.533591 7254 consensus_queue.cc:260] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.533643 7321 consensus_queue.cc:260] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.535908 7254 ts_tablet_manager.cc:1398] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:48.536695 7320 tablet_bootstrap.cc:492] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:48.540347 7321 ts_tablet_manager.cc:1398] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:00:48.546952 7320 tablet_bootstrap.cc:654] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.548902 7320 tablet_bootstrap.cc:492] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:48.549108 7320 ts_tablet_manager.cc:1367] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 14:00:48.549876 7320 raft_consensus.cc:348] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.550120 7320 raft_consensus.cc:374] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.550216 7320 raft_consensus.cc:729] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.550473 7320 consensus_queue.cc:260] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.555155 7320 ts_tablet_manager.cc:1398] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:48.566072 7328 ts_tablet_manager.cc:902] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:48.577062 7328 tablet_copy_client.cc:308] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:48.577824 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet fd3f8661a13944098a18d3eea7ddeb00 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:49081)
I20240501 14:00:48.578007 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet fd3f8661a13944098a18d3eea7ddeb00 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:49081: session id = f8c60f2bbac541dcba4424ab726f9ad9-fd3f8661a13944098a18d3eea7ddeb00
I20240501 14:00:48.580493 2428 tablet_copy_source_session.cc:205] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:48.581938 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f8661a13944098a18d3eea7ddeb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.588573 7328 tablet_copy_client.cc:783] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:48.588853 7328 tablet_copy_client.cc:647] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:48.590775 7328 tablet_copy_client.cc:523] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:48.594321 7328 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:48.635939 7328 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:48.636482 7328 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:00:48.636772 7328 ts_tablet_manager.cc:1367] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.000s
I20240501 14:00:48.637660 7328 raft_consensus.cc:348] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:48.637940 7328 raft_consensus.cc:729] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:00:48.638190 7328 consensus_queue.cc:260] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:00:48.638811 7328 ts_tablet_manager.cc:1398] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:48.639768 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-fd3f8661a13944098a18d3eea7ddeb00 received from {username='slave'} at 127.28.17.5:49081
I20240501 14:00:48.639906 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-fd3f8661a13944098a18d3eea7ddeb00 on tablet fd3f8661a13944098a18d3eea7ddeb00 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:48.653585 6929 consensus_queue.cc:1035] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.657357 7159 raft_consensus.cc:482] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:48.657546 7159 raft_consensus.cc:504] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.658229 7159 leader_election.cc:288] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:48.658697 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6ec775542d46ee901be28099092a63" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:48.658931 5293 raft_consensus.cc:2454] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:48.659341 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6ec775542d46ee901be28099092a63" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:48.659561 4900 raft_consensus.cc:2454] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:48.660019 5099 leader_election.cc:302] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:48.660549 7159 raft_consensus.cc:2781] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:48.660656 7159 raft_consensus.cc:482] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:48.660759 7159 raft_consensus.cc:3037] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.664602 7159 raft_consensus.cc:504] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.666066 7159 leader_election.cc:288] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:48.666141 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6ec775542d46ee901be28099092a63" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:48.666138 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6ec775542d46ee901be28099092a63" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:48.666371 5293 raft_consensus.cc:3037] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.666388 4900 raft_consensus.cc:3037] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:48.669168 4900 raft_consensus.cc:2454] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:48.669308 5293 raft_consensus.cc:2454] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:48.669806 5098 leader_election.cc:302] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:48.671656 7159 raft_consensus.cc:2781] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:48.673758 7159 raft_consensus.cc:686] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:48.674192 7159 consensus_queue.cc:237] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:48.684710 6947 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.695446 2005 catalog_manager.cc:5484] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:48.708242 6929 consensus_queue.cc:1035] T 0724625a6a8e4ce38e8087472b856c5c P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.729773 6764 consensus_queue.cc:1035] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.729869 7157 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.737295 7374 ts_tablet_manager.cc:902] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:48.773973 6764 consensus_queue.cc:1035] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.780287 7374 tablet_copy_client.cc:308] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:48.780961 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet 457c62e1ddf547ad900ed529b3244474 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:41439)
I20240501 14:00:48.781169 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet 457c62e1ddf547ad900ed529b3244474 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:41439: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-457c62e1ddf547ad900ed529b3244474
I20240501 14:00:48.784251 2428 tablet_copy_source_session.cc:205] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:48.787159 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457c62e1ddf547ad900ed529b3244474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.793821 7374 tablet_copy_client.cc:783] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:48.794100 7374 tablet_copy_client.cc:647] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:48.796123 7374 tablet_copy_client.cc:523] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:48.799358 7374 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:48.800163 2538 consensus_queue.cc:237] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:48.802529 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:48.804545 2408 raft_consensus.cc:1261] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:48.805064 2274 raft_consensus.cc:1261] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:48.805776 7189 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:48.806406 7189 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:48.810421 2473 consensus_peers.cc:485] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ee13db8a347476fbdabdb402c93b584. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:48.814924 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:48.816344 5142 tablet_service.cc:1467] Processing CreateTablet for tablet ea342bfcde3b4daa8dcc39126fd42cdc (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.816816 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea342bfcde3b4daa8dcc39126fd42cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.816987 5143 tablet_service.cc:1467] Processing CreateTablet for tablet e8e883d12de74534a02ca9ccaada086f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.816814 7189 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:48.817380 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e883d12de74534a02ca9ccaada086f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.819774 4881 tablet_service.cc:1467] Processing CreateTablet for tablet ea342bfcde3b4daa8dcc39126fd42cdc (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.820258 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea342bfcde3b4daa8dcc39126fd42cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.822799 5014 tablet_service.cc:1467] Processing CreateTablet for tablet e8e883d12de74534a02ca9ccaada086f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.823239 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e883d12de74534a02ca9ccaada086f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.824198 7320 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:48.825599 7320 tablet_bootstrap.cc:654] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.826267 7321 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:48.826190 5013 tablet_service.cc:1467] Processing CreateTablet for tablet ea342bfcde3b4daa8dcc39126fd42cdc (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.826643 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea342bfcde3b4daa8dcc39126fd42cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.828564 7321 tablet_bootstrap.cc:654] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.833515 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 9b91e5a14d82419cb295439f577d2786 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.834097 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b91e5a14d82419cb295439f577d2786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.835278 7321 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:48.836414 7321 ts_tablet_manager.cc:1367] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:00:48.840914 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 9b91e5a14d82419cb295439f577d2786 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.841542 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b91e5a14d82419cb295439f577d2786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.844624 2516 tablet_service.cc:1467] Processing CreateTablet for tablet e8e883d12de74534a02ca9ccaada086f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.844731 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 9b91e5a14d82419cb295439f577d2786 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:48.844686 7321 raft_consensus.cc:348] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.844940 7321 raft_consensus.cc:374] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.845011 7321 raft_consensus.cc:729] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.845082 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e883d12de74534a02ca9ccaada086f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.845283 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b91e5a14d82419cb295439f577d2786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:48.845266 7321 consensus_queue.cc:260] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.845083 2408 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:48.846233 7321 ts_tablet_manager.cc:1398] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 14:00:48.850993 2274 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:48.859752 2002 catalog_manager.cc:5484] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:48.867221 7399 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:48.871026 7399 tablet_bootstrap.cc:654] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.873003 7320 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:48.873353 7320 ts_tablet_manager.cc:1367] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.006s
I20240501 14:00:48.874313 7320 raft_consensus.cc:348] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.874644 7320 raft_consensus.cc:374] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.874780 7320 raft_consensus.cc:729] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.875092 7320 consensus_queue.cc:260] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.876895 7320 ts_tablet_manager.cc:1398] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:48.877252 7320 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:48.880683 7320 tablet_bootstrap.cc:654] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.882105 7399 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:48.882485 7399 ts_tablet_manager.cc:1367] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 14:00:48.883610 7399 raft_consensus.cc:348] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.885183 7398 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:48.885494 7399 raft_consensus.cc:374] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.885669 7399 raft_consensus.cc:729] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.887004 4901 raft_consensus.cc:1261] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.887142 7399 consensus_queue.cc:260] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.887733 7363 consensus_queue.cc:1035] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.889425 5293 raft_consensus.cc:1261] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.891762 7399 ts_tablet_manager.cc:1398] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 14:00:48.896358 7363 consensus_queue.cc:1035] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.896993 7399 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:48.897904 7398 tablet_bootstrap.cc:654] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.901372 7399 tablet_bootstrap.cc:654] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.911929 7399 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:48.912230 7399 ts_tablet_manager.cc:1367] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20240501 14:00:48.913161 7399 raft_consensus.cc:348] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.915691 7399 raft_consensus.cc:374] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.917125 7398 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:48.917369 7398 ts_tablet_manager.cc:1367] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20240501 14:00:48.918824 7398 raft_consensus.cc:348] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.919167 7398 raft_consensus.cc:374] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.919417 7398 raft_consensus.cc:729] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.919744 7398 consensus_queue.cc:260] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.920624 7398 ts_tablet_manager.cc:1398] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:48.920985 7398 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:48.916018 7399 raft_consensus.cc:729] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.922969 7398 tablet_bootstrap.cc:654] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.923415 7399 consensus_queue.cc:260] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.927778 7399 ts_tablet_manager.cc:1398] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:00:48.934772 7399 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:48.936563 7399 tablet_bootstrap.cc:654] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:48.951606 4900 consensus_queue.cc:237] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:00:48.955873 1997 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 0d6ec775542d46ee901be28099092a63 (table rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]). Expected version 1 got 0
I20240501 14:00:48.959843 7414 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:48.959990 7399 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:48.960153 7399 ts_tablet_manager.cc:1367] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20240501 14:00:48.960690 7414 tablet.cc:1620] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:48.961139 7399 raft_consensus.cc:348] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.962275 7399 raft_consensus.cc:374] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.962481 7399 raft_consensus.cc:729] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.964048 7415 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:48.964776 7415 tablet.cc:1620] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:48.965698 7399 consensus_queue.cc:260] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.966822 7398 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:48.967049 7398 ts_tablet_manager.cc:1367] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.003s
I20240501 14:00:48.967929 2002 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 0d6ec775542d46ee901be28099092a63 (table rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]). Expected version 1 got 0
I20240501 14:00:48.972080 2005 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 0d6ec775542d46ee901be28099092a63 (table rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]). Expected version 1 got 0
W20240501 14:00:48.972432 4834 consensus_peers.cc:485] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa9e01283eb491abde0417ffbaff56f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:48.973765 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5] alter complete (version 1)
I20240501 14:00:48.973929 5164 raft_consensus.cc:1261] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.969027 7398 raft_consensus.cc:348] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.974314 7398 raft_consensus.cc:374] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.974463 7398 raft_consensus.cc:729] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.974794 7398 consensus_queue.cc:260] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:48.975383 7398 ts_tablet_manager.cc:1398] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:00:48.972878 5034 raft_consensus.cc:1261] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.979250 7155 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 14:00:48.979972 7157 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:48.980829 7399 ts_tablet_manager.cc:1398] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240501 14:00:48.984421 7157 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:00:48.987123 5032 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:00:48.990700 7416 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:48.991261 7416 tablet.cc:1620] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:48.994808 7320 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:48.994977 7320 ts_tablet_manager.cc:1367] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.118s user 0.015s sys 0.038s
I20240501 14:00:48.995234 5164 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:00:48.995693 1997 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 0d6ec775542d46ee901be28099092a63 (table rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]). Expected version 1 got 0
I20240501 14:00:48.996609 7320 raft_consensus.cc:348] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.996798 7320 raft_consensus.cc:374] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:48.996876 7320 raft_consensus.cc:729] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:48.997120 7320 consensus_queue.cc:260] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:48.997776 7320 ts_tablet_manager.cc:1398] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:48.998052 7320 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:48.999799 7320 tablet_bootstrap.cc:654] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.001813 7320 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:49.002070 7320 ts_tablet_manager.cc:1367] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:49.002949 7320 raft_consensus.cc:348] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.003283 7320 raft_consensus.cc:374] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.003430 7320 raft_consensus.cc:729] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.003756 7320 consensus_queue.cc:260] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.004779 7320 ts_tablet_manager.cc:1398] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:49.015241 1999 catalog_manager.cc:5484] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:00:49.017283 7374 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:49.021901 7374 tablet_bootstrap.cc:492] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:49.022356 7374 ts_tablet_manager.cc:1367] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.223s user 0.048s sys 0.007s
I20240501 14:00:49.023267 7374 raft_consensus.cc:348] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:49.023635 7374 raft_consensus.cc:729] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:49.023895 7374 consensus_queue.cc:260] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:49.024622 7374 ts_tablet_manager.cc:1398] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:49.026940 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-457c62e1ddf547ad900ed529b3244474 received from {username='slave'} at 127.28.17.6:41439
I20240501 14:00:49.027109 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-457c62e1ddf547ad900ed529b3244474 on tablet 457c62e1ddf547ad900ed529b3244474 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:49.027740 7159 raft_consensus.cc:482] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.027948 7159 raft_consensus.cc:504] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.028610 7159 leader_election.cc:288] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:49.029953 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e883d12de74534a02ca9ccaada086f" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:49.030208 2538 raft_consensus.cc:2454] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:49.030781 5097 leader_election.cc:302] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:49.031144 7224 raft_consensus.cc:2781] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.031224 7224 raft_consensus.cc:482] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.031302 7224 raft_consensus.cc:3037] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.035333 7224 raft_consensus.cc:504] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.037968 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e883d12de74534a02ca9ccaada086f" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:49.038233 2538 raft_consensus.cc:3037] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.041157 2538 raft_consensus.cc:2454] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:49.042331 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e883d12de74534a02ca9ccaada086f" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:49.042587 5034 raft_consensus.cc:3037] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.045261 5034 raft_consensus.cc:2454] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:49.046144 5099 leader_election.cc:302] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:49.046706 7363 raft_consensus.cc:2781] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.046873 7363 raft_consensus.cc:686] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:49.047214 7363 consensus_queue.cc:237] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.048969 7224 leader_election.cc:288] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:49.049882 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e883d12de74534a02ca9ccaada086f" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:49.052984 5032 raft_consensus.cc:2362] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1. Re-sending same reply.
I20240501 14:00:49.053236 2002 catalog_manager.cc:5484] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:49.079288 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:49.086356 4901 raft_consensus.cc:1203] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:49.089777 7414 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:49.090351 7414 tablet.cc:1620] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:49.091154 7416 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:49.091603 7416 tablet.cc:1620] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:49.100795 6927 raft_consensus.cc:482] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.101007 6927 raft_consensus.cc:504] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.101727 6927 leader_election.cc:288] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:49.104207 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea342bfcde3b4daa8dcc39126fd42cdc" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:49.104465 5164 raft_consensus.cc:2454] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:49.105089 4967 leader_election.cc:302] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:49.106267 6927 raft_consensus.cc:2781] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.106371 6927 raft_consensus.cc:482] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.106262 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea342bfcde3b4daa8dcc39126fd42cdc" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:49.106539 4901 raft_consensus.cc:2454] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:49.106468 6927 raft_consensus.cc:3037] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.106976 7415 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:49.107395 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=75cc7fc399ee4a4eabbd55e01ea5ddc5] alter complete (version 2)
I20240501 14:00:49.107437 7415 tablet.cc:1620] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:49.109227 6927 raft_consensus.cc:504] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.110030 6927 leader_election.cc:288] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:49.111414 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea342bfcde3b4daa8dcc39126fd42cdc" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:49.111655 4900 raft_consensus.cc:3037] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.113673 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea342bfcde3b4daa8dcc39126fd42cdc" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:49.113890 5164 raft_consensus.cc:3037] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.114333 4900 raft_consensus.cc:2454] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:49.116369 5164 raft_consensus.cc:2454] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:49.116812 4967 leader_election.cc:302] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:49.117244 6653 raft_consensus.cc:2781] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.117463 6653 raft_consensus.cc:686] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:49.117753 6653 consensus_queue.cc:237] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.124400 2005 catalog_manager.cc:5484] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:49.165865 7388 raft_consensus.cc:482] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.166078 7388 raft_consensus.cc:504] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.166926 7388 leader_election.cc:288] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:49.167275 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b91e5a14d82419cb295439f577d2786" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:49.167416 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b91e5a14d82419cb295439f577d2786" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:49.167483 5164 raft_consensus.cc:2454] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:49.167650 5032 raft_consensus.cc:2454] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:49.167912 2471 leader_election.cc:302] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:49.168179 7388 raft_consensus.cc:2781] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.168292 7388 raft_consensus.cc:482] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.168393 7388 raft_consensus.cc:3037] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.170375 7388 raft_consensus.cc:504] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.170956 7388 leader_election.cc:288] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:49.171499 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b91e5a14d82419cb295439f577d2786" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:49.171504 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b91e5a14d82419cb295439f577d2786" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:49.171689 5164 raft_consensus.cc:3037] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.171725 5032 raft_consensus.cc:3037] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.174084 5164 raft_consensus.cc:2454] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:49.174283 5032 raft_consensus.cc:2454] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:49.174670 2471 leader_election.cc:302] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:49.175004 7388 raft_consensus.cc:2781] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.175156 7388 raft_consensus.cc:686] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:49.175511 7388 consensus_queue.cc:237] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:49.178897 2005 catalog_manager.cc:5484] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:49.216954 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:49.224830 7414 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:49.225515 7415 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:49.226259 7416 tablet.cc:1722] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:49.235379 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=75cc7fc399ee4a4eabbd55e01ea5ddc5] alter complete (version 3)
I20240501 14:00:49.337328 1999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:49.337585 1999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:49.341125 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 0d6ec775542d46ee901be28099092a63
I20240501 14:00:49.342257 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6ec775542d46ee901be28099092a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:49.342470 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6ec775542d46ee901be28099092a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:49.342689 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6ec775542d46ee901be28099092a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:49 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:49.343713 7451 tablet_replica.cc:330] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:49.344013 7451 raft_consensus.cc:2229] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:49.344311 7451 raft_consensus.cc:2258] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.345214 7451 ts_tablet_manager.cc:1875] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.350466 5032 raft_consensus.cc:1203] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:49.353391 7451 ts_tablet_manager.cc:1888] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.353628 7451 log.cc:1196] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/0d6ec775542d46ee901be28099092a63
I20240501 14:00:49.354197 7451 ts_tablet_manager.cc:1909] T 0d6ec775542d46ee901be28099092a63 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:49.355443 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 0d6ec775542d46ee901be28099092a63 (table rebalancer_dynamic_table_new_name [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]) successfully deleted
I20240501 14:00:49.358386 7450 tablet_replica.cc:330] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:49.358476 7449 tablet_replica.cc:330] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:49.358664 7450 raft_consensus.cc:2229] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:49.358719 7449 raft_consensus.cc:2229] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:49.358909 7450 raft_consensus.cc:2258] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.359069 7449 raft_consensus.cc:2258] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.359889 7449 ts_tablet_manager.cc:1875] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.362715 7450 ts_tablet_manager.cc:1875] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:49.365298 7449 ts_tablet_manager.cc:1888] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.365429 7449 log.cc:1196] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/0d6ec775542d46ee901be28099092a63
I20240501 14:00:49.365811 7449 ts_tablet_manager.cc:1909] T 0d6ec775542d46ee901be28099092a63 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:49.366858 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 0d6ec775542d46ee901be28099092a63 (table rebalancer_dynamic_table_new_name [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]) successfully deleted
I20240501 14:00:49.372277 7450 ts_tablet_manager.cc:1888] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:49.372439 7450 log.cc:1196] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/0d6ec775542d46ee901be28099092a63
I20240501 14:00:49.372970 7450 ts_tablet_manager.cc:1909] T 0d6ec775542d46ee901be28099092a63 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:49.374303 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 0d6ec775542d46ee901be28099092a63 (table rebalancer_dynamic_table_new_name [id=75cc7fc399ee4a4eabbd55e01ea5ddc5]) successfully deleted
I20240501 14:00:49.416682 7374 ts_tablet_manager.cc:902] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:49.417788 7374 tablet_copy_client.cc:308] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:49.418466 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 2ee13db8a347476fbdabdb402c93b584 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:45411)
I20240501 14:00:49.418639 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 2ee13db8a347476fbdabdb402c93b584 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:45411: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-2ee13db8a347476fbdabdb402c93b584
I20240501 14:00:49.421020 2558 tablet_copy_source_session.cc:205] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:49.422259 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee13db8a347476fbdabdb402c93b584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.427457 7374 tablet_copy_client.cc:783] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:49.427691 7374 tablet_copy_client.cc:647] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:49.429193 7374 tablet_copy_client.cc:523] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:49.432328 7374 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:49.443811 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:49.453029 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb]), partition=
I20240501 14:00:49.453017 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb]), partition=
I20240501 14:00:49.453464 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd7b6f4d60b4c68ba7c042054b096c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.453467 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd7b6f4d60b4c68ba7c042054b096c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.455125 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb]), partition=
I20240501 14:00:49.455513 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd7b6f4d60b4c68ba7c042054b096c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.463680 7473 tablet_bootstrap.cc:492] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:49.465502 7473 tablet_bootstrap.cc:654] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.469236 7474 tablet_bootstrap.cc:492] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:49.471629 7474 tablet_bootstrap.cc:654] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.478299 7374 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:49.479204 7374 tablet_bootstrap.cc:492] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:49.479506 7374 ts_tablet_manager.cc:1367] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.047s user 0.043s sys 0.003s
I20240501 14:00:49.480360 7374 raft_consensus.cc:348] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:49.480624 7374 raft_consensus.cc:729] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:49.480882 7374 consensus_queue.cc:260] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:49.481642 7374 ts_tablet_manager.cc:1398] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:49.482472 7476 tablet_bootstrap.cc:492] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:49.482806 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-2ee13db8a347476fbdabdb402c93b584 received from {username='slave'} at 127.28.17.6:45411
I20240501 14:00:49.482950 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-2ee13db8a347476fbdabdb402c93b584 on tablet 2ee13db8a347476fbdabdb402c93b584 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:49.484215 7476 tablet_bootstrap.cc:654] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.497059 7473 tablet_bootstrap.cc:492] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:49.497212 7224 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.497356 7473 ts_tablet_manager.cc:1367] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.005s
I20240501 14:00:49.498062 6927 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.498185 7473 raft_consensus.cc:348] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.498397 7473 raft_consensus.cc:374] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.498519 7473 raft_consensus.cc:729] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.498778 7473 consensus_queue.cc:260] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.516139 7473 ts_tablet_manager.cc:1398] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:00:49.521659 7474 tablet_bootstrap.cc:492] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:49.521855 7474 ts_tablet_manager.cc:1367] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.015s
I20240501 14:00:49.522770 7474 raft_consensus.cc:348] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.523075 7474 raft_consensus.cc:374] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.523202 7474 raft_consensus.cc:729] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.523494 7474 consensus_queue.cc:260] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.523640 7481 ts_tablet_manager.cc:902] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:49.524616 7363 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.525650 7474 ts_tablet_manager.cc:1398] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:49.527863 7476 tablet_bootstrap.cc:492] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:49.528095 7476 ts_tablet_manager.cc:1367] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.023s
I20240501 14:00:49.530205 7476 raft_consensus.cc:348] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.530540 7476 raft_consensus.cc:374] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.530678 7476 raft_consensus.cc:729] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.530956 7476 consensus_queue.cc:260] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.531562 7476 ts_tablet_manager.cc:1398] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:49.533875 6929 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.561069 7397 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.586163 7481 tablet_copy_client.cc:308] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:00:49.587213 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet 1fa9e01283eb491abde0417ffbaff56f from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:38989)
I20240501 14:00:49.587416 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet 1fa9e01283eb491abde0417ffbaff56f from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:38989: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-1fa9e01283eb491abde0417ffbaff56f
I20240501 14:00:49.587760 7388 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.591009 4921 tablet_copy_source_session.cc:205] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:49.597445 7481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9e01283eb491abde0417ffbaff56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.604238 7481 tablet_copy_client.cc:783] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:49.604631 7481 tablet_copy_client.cc:647] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:49.606628 7481 tablet_copy_client.cc:523] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:49.609678 7481 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:49.611579 7279 raft_consensus.cc:1050] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:00:49.612331 7279 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:49.616371 7481 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:49.616768 2274 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:49.616850 7481 tablet_bootstrap.cc:492] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:00:49.617107 7481 ts_tablet_manager.cc:1367] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240501 14:00:49.617281 7169 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:49.617923 7481 raft_consensus.cc:348] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:00:49.618170 7481 raft_consensus.cc:729] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:00:49.618417 7481 consensus_queue.cc:260] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:00:49.619078 7481 ts_tablet_manager.cc:1398] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 14:00:49.620059 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-1fa9e01283eb491abde0417ffbaff56f received from {username='slave'} at 127.28.17.2:38989
I20240501 14:00:49.620186 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-1fa9e01283eb491abde0417ffbaff56f on tablet 1fa9e01283eb491abde0417ffbaff56f with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:00:49.622671 4900 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:49.623229 2146 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:49.623953 7169 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:49.624576 7169 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:49.637440 2275 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.636485 7169 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.646939 2002 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.652411 4900 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.654435 2146 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.671697 2408 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:00:49.674386 2146 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:49.675042 7436 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:49.677634 7279 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.681921 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd3f8661a13944098a18d3eea7ddeb00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:49.685082 1997 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:49.689797 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet fd3f8661a13944098a18d3eea7ddeb00 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:49.709518 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:49.713559 2146 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.718997 4900 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:49.719586 7169 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:49.721731 4900 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:00:49.722357 7504 tablet_replica.cc:330] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:49.722648 7504 raft_consensus.cc:2229] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:49.722965 7504 raft_consensus.cc:2258] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:49.724411 7504 ts_tablet_manager.cc:1875] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:49.730849 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2f1ffd51584255b020ed742f30f573 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.731346 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2f1ffd51584255b020ed742f30f573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.732182 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2f1ffd51584255b020ed742f30f573 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.732569 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2f1ffd51584255b020ed742f30f573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.733441 7504 ts_tablet_manager.cc:1888] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:49.733561 7504 log.cc:1196] T fd3f8661a13944098a18d3eea7ddeb00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/fd3f8661a13944098a18d3eea7ddeb00
I20240501 14:00:49.734408 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet fd3f8661a13944098a18d3eea7ddeb00 (table rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]) successfully deleted
I20240501 14:00:49.735036 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2f1ffd51584255b020ed742f30f573 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.735538 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2f1ffd51584255b020ed742f30f573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.738265 7514 tablet_bootstrap.cc:492] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:49.739683 7514 tablet_bootstrap.cc:654] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.739677 5274 tablet_service.cc:1467] Processing CreateTablet for tablet b911b0ff6ef446d8a6637edca57e570d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.740098 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b911b0ff6ef446d8a6637edca57e570d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.742969 5143 tablet_service.cc:1467] Processing CreateTablet for tablet b911b0ff6ef446d8a6637edca57e570d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.743659 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b911b0ff6ef446d8a6637edca57e570d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.746372 7474 tablet_bootstrap.cc:492] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:49.748103 7474 tablet_bootstrap.cc:654] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.756695 5012 tablet_service.cc:1467] Processing CreateTablet for tablet b911b0ff6ef446d8a6637edca57e570d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.757228 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b911b0ff6ef446d8a6637edca57e570d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.757279 7517 tablet_bootstrap.cc:492] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:49.757768 5276 tablet_service.cc:1467] Processing CreateTablet for tablet beb5db06e4b74430b2bcbbecc82796e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.758214 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5db06e4b74430b2bcbbecc82796e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.759562 7517 tablet_bootstrap.cc:654] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.760133 7214 raft_consensus.cc:482] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.760270 7214 raft_consensus.cc:504] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.760535 7474 tablet_bootstrap.cc:492] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:49.760701 7474 ts_tablet_manager.cc:1367] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20240501 14:00:49.761662 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7b6f4d60b4c68ba7c042054b096c0" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:49.761641 7474 raft_consensus.cc:348] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.761860 7474 raft_consensus.cc:374] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.761896 4901 raft_consensus.cc:2454] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:49.761935 7474 raft_consensus.cc:729] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.762176 7474 consensus_queue.cc:260] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.762813 7474 ts_tablet_manager.cc:1398] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:00:49.763074 5142 tablet_service.cc:1467] Processing CreateTablet for tablet beb5db06e4b74430b2bcbbecc82796e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.763137 7214 leader_election.cc:288] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:49.763525 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5db06e4b74430b2bcbbecc82796e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.763597 5229 leader_election.cc:302] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:49.765301 7517 tablet_bootstrap.cc:492] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:49.765350 7214 raft_consensus.cc:2781] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.765456 7517 ts_tablet_manager.cc:1367] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240501 14:00:49.765528 7214 raft_consensus.cc:482] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.765614 7214 raft_consensus.cc:3037] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.767824 7517 raft_consensus.cc:348] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.768571 7517 raft_consensus.cc:374] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.768577 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7b6f4d60b4c68ba7c042054b096c0" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:49.768663 7517 raft_consensus.cc:729] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.768793 2274 raft_consensus.cc:2454] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:49.769346 7517 consensus_queue.cc:260] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.773511 7214 raft_consensus.cc:504] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.774418 7517 ts_tablet_manager.cc:1398] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240501 14:00:49.774801 7214 leader_election.cc:288] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:49.775774 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7b6f4d60b4c68ba7c042054b096c0" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:49.776012 2274 raft_consensus.cc:3037] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.782358 7476 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:49.783933 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd7b6f4d60b4c68ba7c042054b096c0" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:49.784134 7476 tablet_bootstrap.cc:654] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.784160 4901 raft_consensus.cc:3037] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.789386 2274 raft_consensus.cc:2454] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:49.789963 5228 leader_election.cc:302] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:49.790665 4901 raft_consensus.cc:2454] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:49.790318 7214 raft_consensus.cc:2781] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.791474 4881 tablet_service.cc:1467] Processing CreateTablet for tablet beb5db06e4b74430b2bcbbecc82796e2 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:49.792989 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5db06e4b74430b2bcbbecc82796e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:49.805634 7214 raft_consensus.cc:686] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:49.806078 7214 consensus_queue.cc:237] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.812366 7474 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:49.817035 7474 tablet_bootstrap.cc:654] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.823175 1999 catalog_manager.cc:5484] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:49.824316 7514 tablet_bootstrap.cc:492] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:49.824460 7514 ts_tablet_manager.cc:1367] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.086s user 0.028s sys 0.025s
I20240501 14:00:49.825088 7514 raft_consensus.cc:348] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.825269 7514 raft_consensus.cc:374] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.825322 7514 raft_consensus.cc:729] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.825517 7514 consensus_queue.cc:260] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.825711 7476 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:49.825996 7514 ts_tablet_manager.cc:1398] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:00:49.826006 7476 ts_tablet_manager.cc:1367] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.004s
I20240501 14:00:49.828619 7474 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:49.828895 7474 ts_tablet_manager.cc:1367] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240501 14:00:49.829818 7474 raft_consensus.cc:348] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.830382 7520 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:49.831041 7476 raft_consensus.cc:348] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.831270 7474 raft_consensus.cc:374] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.831413 7474 raft_consensus.cc:729] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.831650 7476 raft_consensus.cc:374] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.831660 7474 consensus_queue.cc:260] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.832031 7476 raft_consensus.cc:729] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.832242 7474 ts_tablet_manager.cc:1398] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:49.835186 7476 consensus_queue.cc:260] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.836539 7520 tablet_bootstrap.cc:654] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.840024 7476 ts_tablet_manager.cc:1398] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:00:49.840373 7476 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:49.841992 7476 tablet_bootstrap.cc:654] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.833003 7523 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:49.845925 7523 tablet_bootstrap.cc:654] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.845974 7169 raft_consensus.cc:1050] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:49.846810 7169 consensus_queue.cc:237] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:49.849401 5032 raft_consensus.cc:1261] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:49.849429 2538 raft_consensus.cc:1261] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:49.849596 2146 raft_consensus.cc:1261] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:49.850263 7436 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:49.852012 7520 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:49.851907 7279 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:49.852437 7279 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:49.853899 7520 ts_tablet_manager.cc:1367] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240501 14:00:49.856595 7520 raft_consensus.cc:348] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.856842 7520 raft_consensus.cc:374] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.856921 7520 raft_consensus.cc:729] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.857169 7520 consensus_queue.cc:260] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.860340 7520 ts_tablet_manager.cc:1398] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:49.863021 7476 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:49.863322 7476 ts_tablet_manager.cc:1367] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240501 14:00:49.865339 7520 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:49.866547 7476 raft_consensus.cc:348] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.866866 7476 raft_consensus.cc:374] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.867051 7476 raft_consensus.cc:729] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.867406 7476 consensus_queue.cc:260] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.868902 7520 tablet_bootstrap.cc:654] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:49.869308 7476 ts_tablet_manager.cc:1398] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:49.872299 7523 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:49.872488 7523 ts_tablet_manager.cc:1367] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.003s
I20240501 14:00:49.873409 7520 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:49.873361 7523 raft_consensus.cc:348] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.873575 7523 raft_consensus.cc:374] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.873667 7523 raft_consensus.cc:729] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.873874 7523 consensus_queue.cc:260] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.874101 7520 ts_tablet_manager.cc:1367] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:00:49.874425 7523 ts_tablet_manager.cc:1398] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:49.874953 7520 raft_consensus.cc:348] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.875154 7520 raft_consensus.cc:374] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:49.875221 7520 raft_consensus.cc:729] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:49.875542 7520 consensus_queue.cc:260] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.876228 7520 ts_tablet_manager.cc:1398] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:49.903738 6947 raft_consensus.cc:482] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.903946 6947 raft_consensus.cc:504] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.904639 6947 leader_election.cc:288] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:49.906190 7169 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.907199 2146 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.909003 5034 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.910308 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2f1ffd51584255b020ed742f30f573" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:49.910992 2143 raft_consensus.cc:2454] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:49.911620 4834 leader_election.cc:302] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:49.912233 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2f1ffd51584255b020ed742f30f573" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:49.912425 2406 raft_consensus.cc:2454] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:49.912802 7214 raft_consensus.cc:482] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.912959 7214 raft_consensus.cc:504] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.913627 7214 leader_election.cc:288] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:49.915951 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b911b0ff6ef446d8a6637edca57e570d" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:49.916225 5164 raft_consensus.cc:2454] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:49.916321 2538 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.917539 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b911b0ff6ef446d8a6637edca57e570d" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:49.917749 5033 raft_consensus.cc:2454] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:49.918521 5230 leader_election.cc:302] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 6f81f803ec7746ccb502c59b7cbf8eb2; no voters:
I20240501 14:00:49.918848 7157 raft_consensus.cc:2781] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.918965 7157 raft_consensus.cc:482] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.923586 2002 catalog_manager.cc:5484] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.923843 7522 raft_consensus.cc:2781] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.923954 7522 raft_consensus.cc:482] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.924053 7522 raft_consensus.cc:3037] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.927362 7522 raft_consensus.cc:504] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:49.930385 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b911b0ff6ef446d8a6637edca57e570d" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:49.930644 5033 raft_consensus.cc:3037] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.930632 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b911b0ff6ef446d8a6637edca57e570d" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:49.930884 5164 raft_consensus.cc:3037] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.932868 5164 raft_consensus.cc:2454] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:49.933328 5033 raft_consensus.cc:2454] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:49.934011 5230 leader_election.cc:302] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 6f81f803ec7746ccb502c59b7cbf8eb2; no voters:
I20240501 14:00:49.934434 7214 raft_consensus.cc:2781] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.936889 7522 leader_election.cc:288] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:49.939555 2406 consensus_queue.cc:237] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:49.944350 2146 raft_consensus.cc:1261] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:49.945003 7532 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:49.946444 5033 raft_consensus.cc:1261] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:49.946599 7157 raft_consensus.cc:3037] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.948199 7532 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.949880 7157 raft_consensus.cc:504] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.951259 7436 consensus_queue.cc:1035] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:49.952704 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:49.954831 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 457c62e1ddf547ad900ed529b3244474 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:49.955623 5034 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.955861 2146 raft_consensus.cc:2932] T 457c62e1ddf547ad900ed529b3244474 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:49.959838 7157 leader_election.cc:288] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:49.961017 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2f1ffd51584255b020ed742f30f573" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:49.961257 2143 raft_consensus.cc:3037] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.961341 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2f1ffd51584255b020ed742f30f573" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:49.961539 2406 raft_consensus.cc:3037] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:49.964206 2406 raft_consensus.cc:2454] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:49.967662 4900 raft_consensus.cc:1261] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:49.969066 2143 raft_consensus.cc:2454] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:49.969393 7546 consensus_queue.cc:1035] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.969889 5033 raft_consensus.cc:1203] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:49.972590 2274 raft_consensus.cc:1261] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:49.973160 7546 consensus_queue.cc:1035] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:49.973484 4834 leader_election.cc:302] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:49.974370 6947 raft_consensus.cc:2781] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:49.975318 6947 raft_consensus.cc:686] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:49.975638 6947 consensus_queue.cc:237] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.975859 2274 raft_consensus.cc:1203] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:49.958868 1997 catalog_manager.cc:5484] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:49.986373 2005 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (table rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb]). Expected version 1 got 0
I20240501 14:00:49.990103 2516 tablet_service.cc:1514] Processing DeleteTablet for tablet 457c62e1ddf547ad900ed529b3244474 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:49.996006 7522 raft_consensus.cc:482] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:49.996230 7522 raft_consensus.cc:504] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:49.996977 7522 leader_election.cc:288] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:49.997624 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5db06e4b74430b2bcbbecc82796e2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:49.997768 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5db06e4b74430b2bcbbecc82796e2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:49.997922 4900 raft_consensus.cc:2454] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:49.998106 5164 raft_consensus.cc:2454] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:49.998605 5229 leader_election.cc:302] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:49.998965 7522 raft_consensus.cc:2781] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:49.999053 7522 raft_consensus.cc:482] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:49.999150 7522 raft_consensus.cc:3037] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.001952 7522 raft_consensus.cc:504] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:50.009768 7552 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:50.009853 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5db06e4b74430b2bcbbecc82796e2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:50.010167 4899 raft_consensus.cc:3037] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.011011 7552 tablet.cc:1620] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:50.013002 4899 raft_consensus.cc:2454] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:50.013687 5229 leader_election.cc:302] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:50.015482 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5db06e4b74430b2bcbbecc82796e2" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:50.015645 7522 leader_election.cc:288] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:50.015728 5164 raft_consensus.cc:3037] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.018303 7553 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:50.018585 5164 raft_consensus.cc:2454] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:50.018956 7553 tablet.cc:1620] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:50.019793 7546 raft_consensus.cc:2781] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:50.019958 7546 raft_consensus.cc:686] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:50.020316 7546 consensus_queue.cc:237] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:50.035171 7554 tablet_replica.cc:330] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:50.035523 7554 raft_consensus.cc:2229] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.035840 7554 raft_consensus.cc:2258] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.042371 7554 ts_tablet_manager.cc:1875] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:50.043780 7214 raft_consensus.cc:686] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
W20240501 14:00:50.044636 4834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.2:46301, user_credentials={real_user=slave}} blocked reactor thread for 64912.5us
I20240501 14:00:50.044608 7214 consensus_queue.cc:237] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:50.045128 2002 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (table rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb]). Expected version 1 got 0
I20240501 14:00:50.044150 1997 catalog_manager.cc:5484] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:50.053426 7564 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:50.054167 7564 tablet.cc:1620] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:50.053174 2002 catalog_manager.cc:5484] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:50.059128 1999 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (table rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb]). Expected version 1 got 0
I20240501 14:00:50.061749 7554 ts_tablet_manager.cc:1888] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:50.061956 7554 log.cc:1196] T 457c62e1ddf547ad900ed529b3244474 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/457c62e1ddf547ad900ed529b3244474
I20240501 14:00:50.063608 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 457c62e1ddf547ad900ed529b3244474 (table rebalance_test_table_13 [id=43e6d734851d4d129852e2bdbe80131c]) successfully deleted
I20240501 14:00:50.063453 2000 catalog_manager.cc:5484] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:50.065924 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb] alter complete (version 1)
I20240501 14:00:50.089871 2406 consensus_queue.cc:237] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:50.092550 2146 raft_consensus.cc:1261] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:50.092753 2538 raft_consensus.cc:1261] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:00:50.092943 2343 consensus_peers.cc:485] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f67ca7eac95d4a7fbd28667bdde44a22. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.093299 7436 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:50.093597 7532 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:50.096442 7169 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:50.097095 2146 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:50.103555 2000 catalog_manager.cc:5484] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:50.114813 2537 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:50.184013 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:50.191984 7552 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:50.192476 7552 tablet.cc:1620] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:50.192641 7564 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:50.193068 7564 tablet.cc:1620] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:50.197265 7553 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:50.197775 7553 tablet.cc:1620] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:50.202473 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=e09207aa3d1744808acd752a26a10eeb] alter complete (version 2)
I20240501 14:00:50.307569 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:50.314409 7552 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:50.315011 7553 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:50.316958 7564 tablet.cc:1722] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:50.327477 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=e09207aa3d1744808acd752a26a10eeb] alter complete (version 3)
I20240501 14:00:50.412215 7214 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.416333 7558 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.431948 2000 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:50.432242 2000 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:50.432664 7551 raft_consensus.cc:1050] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:00:50.433501 7551 consensus_queue.cc:237] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:00:50.436268 5034 raft_consensus.cc:1261] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:50.436290 5161 raft_consensus.cc:1261] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:50.436455 2275 raft_consensus.cc:1261] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:50.436975 6947 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:50.437249 7157 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:50.437811 7157 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:50.444880 7551 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.448529 2000 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 1bd7b6f4d60b4c68ba7c042054b096c0
I20240501 14:00:50.449515 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:50 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:50.450276 5034 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.453464 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:50 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:50.456200 1997 catalog_manager.cc:5484] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:50.458268 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd7b6f4d60b4c68ba7c042054b096c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:50 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:50.458722 7558 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.466472 2274 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.467356 5161 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.473013 4901 consensus_queue.cc:237] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:00:50.475473 2274 raft_consensus.cc:1261] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:50.476465 7621 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:50.476810 5033 raft_consensus.cc:1261] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:50.477643 6947 consensus_queue.cc:1035] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:50.478744 7619 tablet_replica.cc:330] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:50.478960 6947 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.479462 7617 tablet_replica.cc:330] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:50.478967 7619 raft_consensus.cc:2229] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.480011 7619 raft_consensus.cc:2258] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.479676 7617 raft_consensus.cc:2229] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:50.481635 7617 raft_consensus.cc:2258] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.483137 7214 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.484323 2277 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.490005 7544 consensus_queue.cc:1035] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.496611 7397 raft_consensus.cc:1050] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:50.496788 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fa9e01283eb491abde0417ffbaff56f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:50.497467 7397 consensus_queue.cc:237] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:50.497809 2000 catalog_manager.cc:5484] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.499471 7617 ts_tablet_manager.cc:1875] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:50.503427 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa9e01283eb491abde0417ffbaff56f with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:50.509029 7619 ts_tablet_manager.cc:1875] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:50.510250 7621 consensus_queue.cc:1035] T 1d2f1ffd51584255b020ed742f30f573 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:50.511881 5033 raft_consensus.cc:2932] T 1fa9e01283eb491abde0417ffbaff56f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:00:50.516923 7620 tablet_replica.cc:330] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:50.520032 7617 ts_tablet_manager.cc:1888] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:50.521608 2274 raft_consensus.cc:1261] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:50.522212 5033 raft_consensus.cc:1261] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:50.522418 7388 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:50.522954 7617 log.cc:1196] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/1bd7b6f4d60b4c68ba7c042054b096c0
I20240501 14:00:50.523579 7617 ts_tablet_manager.cc:1909] T 1bd7b6f4d60b4c68ba7c042054b096c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:50.524493 7388 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:50.524755 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (table rebalancer_dynamic_table_new_name [id=e09207aa3d1744808acd752a26a10eeb]) successfully deleted
I20240501 14:00:50.529723 7620 raft_consensus.cc:2229] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.529917 2408 raft_consensus.cc:1261] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:50.530026 7620 raft_consensus.cc:2258] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.530711 7491 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:50.532407 7620 ts_tablet_manager.cc:1875] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:50.549870 7619 ts_tablet_manager.cc:1888] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:50.550019 7619 log.cc:1196] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/1bd7b6f4d60b4c68ba7c042054b096c0
I20240501 14:00:50.553107 7620 ts_tablet_manager.cc:1888] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:50.553249 7620 log.cc:1196] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/1bd7b6f4d60b4c68ba7c042054b096c0
I20240501 14:00:50.553647 7620 ts_tablet_manager.cc:1909] T 1bd7b6f4d60b4c68ba7c042054b096c0 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:50.553822 7619 ts_tablet_manager.cc:1909] T 1bd7b6f4d60b4c68ba7c042054b096c0 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:50.554728 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (table rebalancer_dynamic_table_new_name [id=e09207aa3d1744808acd752a26a10eeb]) successfully deleted
I20240501 14:00:50.554987 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:50.556095 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 1bd7b6f4d60b4c68ba7c042054b096c0 (table rebalancer_dynamic_table_new_name [id=e09207aa3d1744808acd752a26a10eeb]) successfully deleted
I20240501 14:00:50.564504 5142 tablet_service.cc:1467] Processing CreateTablet for tablet bb9921ad9ce549e18d4f7d931074b81e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885]), partition=
I20240501 14:00:50.564936 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9921ad9ce549e18d4f7d931074b81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.566586 5012 tablet_service.cc:1467] Processing CreateTablet for tablet bb9921ad9ce549e18d4f7d931074b81e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885]), partition=
I20240501 14:00:50.567025 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9921ad9ce549e18d4f7d931074b81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.576107 5274 tablet_service.cc:1467] Processing CreateTablet for tablet bb9921ad9ce549e18d4f7d931074b81e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885]), partition=
I20240501 14:00:50.579041 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9921ad9ce549e18d4f7d931074b81e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.588014 7633 tablet_bootstrap.cc:492] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:50.591977 7388 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.592958 2277 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.593640 7626 tablet_replica.cc:330] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:50.593909 7626 raft_consensus.cc:2229] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.594250 7626 raft_consensus.cc:2258] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.594909 7633 tablet_bootstrap.cc:654] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.595515 7626 ts_tablet_manager.cc:1875] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:50.596963 2408 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.607385 1999 catalog_manager.cc:5484] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.614015 7633 tablet_bootstrap.cc:492] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:50.614270 7633 ts_tablet_manager.cc:1367] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20240501 14:00:50.615751 7633 raft_consensus.cc:348] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.616964 7633 raft_consensus.cc:374] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.617139 7633 raft_consensus.cc:729] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.618382 7633 consensus_queue.cc:260] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.620033 5034 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.620340 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:50.622781 7633 ts_tablet_manager.cc:1398] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 14:00:50.635113 7626 ts_tablet_manager.cc:1888] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:00:50.636096 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 878eb076eee94aa8988b3588d5395cba (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.636270 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe1b54933ad446082b1d2840b2c6e49 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.636564 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878eb076eee94aa8988b3588d5395cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.636710 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe1b54933ad446082b1d2840b2c6e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.643712 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe1b54933ad446082b1d2840b2c6e49 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.643854 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 878eb076eee94aa8988b3588d5395cba (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.644181 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe1b54933ad446082b1d2840b2c6e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.644794 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878eb076eee94aa8988b3588d5395cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.641297 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 878eb076eee94aa8988b3588d5395cba (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.648074 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878eb076eee94aa8988b3588d5395cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.648823 2537 consensus_queue.cc:237] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:50.651335 7363 raft_consensus.cc:482] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.651487 7363 raft_consensus.cc:504] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.642882 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe1b54933ad446082b1d2840b2c6e49 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.652451 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe1b54933ad446082b1d2840b2c6e49. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:00:50.654017 5099 leader_election.cc:341] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Not found: Tablet not found: bb9921ad9ce549e18d4f7d931074b81e
I20240501 14:00:50.654564 7363 leader_election.cc:288] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:50.657183 5014 tablet_service.cc:1467] Processing CreateTablet for tablet 36f03024767442ac911a8a44f7ecdceb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.657649 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f03024767442ac911a8a44f7ecdceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.658883 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9921ad9ce549e18d4f7d931074b81e" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:50.659437 7626 log.cc:1196] T 1fa9e01283eb491abde0417ffbaff56f P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/1fa9e01283eb491abde0417ffbaff56f
I20240501 14:00:50.662827 5033 raft_consensus.cc:1261] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:50.663343 7633 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:50.664592 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 36f03024767442ac911a8a44f7ecdceb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.665016 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f03024767442ac911a8a44f7ecdceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.665161 7633 tablet_bootstrap.cc:654] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:50.670219 5099 leader_election.cc:341] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.670428 5099 leader_election.cc:302] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:50.671460 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 36f03024767442ac911a8a44f7ecdceb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:50.671507 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 1fa9e01283eb491abde0417ffbaff56f (table rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]) successfully deleted
I20240501 14:00:50.671886 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f03024767442ac911a8a44f7ecdceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.672151 7363 raft_consensus.cc:2726] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:50.673866 7388 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:50.678342 7640 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:50.682945 7640 tablet_bootstrap.cc:654] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.683709 7634 tablet_bootstrap.cc:492] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:50.686457 7397 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.687284 5031 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.691123 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ee13db8a347476fbdabdb402c93b584 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:50.692804 7640 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:50.693696 7640 ts_tablet_manager.cc:1367] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240501 14:00:50.696323 7640 raft_consensus.cc:348] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:50.696719 1999 catalog_manager.cc:5484] T 2ee13db8a347476fbdabdb402c93b584 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.699750 7640 raft_consensus.cc:374] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.700062 7640 raft_consensus.cc:729] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.700655 7640 consensus_queue.cc:260] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:50.700858 7639 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:50.701457 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ee13db8a347476fbdabdb402c93b584 with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:50.704005 2277 raft_consensus.cc:1261] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:50.706110 7634 tablet_bootstrap.cc:654] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.709877 7645 tablet_replica.cc:330] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:50.710180 7645 raft_consensus.cc:2229] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:50.710711 7640 ts_tablet_manager.cc:1398] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240501 14:00:50.711234 7645 raft_consensus.cc:2258] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:50.716157 7397 consensus_queue.cc:1035] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:50.716624 7645 ts_tablet_manager.cc:1875] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:50.719254 2277 raft_consensus.cc:2932] T 2ee13db8a347476fbdabdb402c93b584 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:50.732014 7645 ts_tablet_manager.cc:1888] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:50.733791 7645 log.cc:1196] T 2ee13db8a347476fbdabdb402c93b584 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/2ee13db8a347476fbdabdb402c93b584
I20240501 14:00:50.734910 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 2ee13db8a347476fbdabdb402c93b584 (table rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]) successfully deleted
I20240501 14:00:50.736052 7641 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:50.741954 7641 tablet_bootstrap.cc:654] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.744261 7639 tablet_bootstrap.cc:654] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.749435 7634 tablet_bootstrap.cc:492] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:50.749720 7634 ts_tablet_manager.cc:1367] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.000s
I20240501 14:00:50.750780 7634 raft_consensus.cc:348] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.751068 7633 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:50.751191 7634 raft_consensus.cc:374] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.751338 7633 ts_tablet_manager.cc:1367] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.088s user 0.012s sys 0.018s
I20240501 14:00:50.751345 7634 raft_consensus.cc:729] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.751585 7634 consensus_queue.cc:260] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.752307 7633 raft_consensus.cc:348] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.752611 7633 raft_consensus.cc:374] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.752732 7633 raft_consensus.cc:729] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.752856 7634 ts_tablet_manager.cc:1398] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:50.753242 6489 raft_consensus.cc:482] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.753391 6489 raft_consensus.cc:504] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:50.754063 6489 leader_election.cc:288] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:50.753027 7633 consensus_queue.cc:260] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.756340 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe1b54933ad446082b1d2840b2c6e49" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:50.756590 7633 ts_tablet_manager.cc:1398] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:00:50.756909 7633 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
W20240501 14:00:50.759430 2080 leader_election.cc:341] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.759903 7633 tablet_bootstrap.cc:654] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.759850 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe1b54933ad446082b1d2840b2c6e49" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
W20240501 14:00:50.762513 2080 leader_election.cc:341] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.762682 2080 leader_election.cc:302] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:50.762976 6489 raft_consensus.cc:2726] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:50.764772 7639 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:50.765853 7634 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
W20240501 14:00:50.766145 4967 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.3:33537, user_credentials={real_user=slave}} blocked reactor thread for 54291.7us
I20240501 14:00:50.768328 7634 tablet_bootstrap.cc:654] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.771906 7633 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:50.772078 7633 ts_tablet_manager.cc:1367] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240501 14:00:50.773008 7633 raft_consensus.cc:348] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:50.773319 7633 raft_consensus.cc:374] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.773502 7633 raft_consensus.cc:729] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.773789 7633 consensus_queue.cc:260] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:50.775701 7633 ts_tablet_manager.cc:1398] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:50.776059 7633 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:50.778627 7633 tablet_bootstrap.cc:654] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.779472 7634 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:50.779992 7634 ts_tablet_manager.cc:1367] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240501 14:00:50.780910 7634 raft_consensus.cc:348] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.781240 7634 raft_consensus.cc:374] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.781428 7634 raft_consensus.cc:729] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.781880 7634 consensus_queue.cc:260] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.787564 7634 ts_tablet_manager.cc:1398] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:50.793277 7594 raft_consensus.cc:482] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.793480 7594 raft_consensus.cc:504] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.794222 7594 leader_election.cc:288] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:50.795188 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9921ad9ce549e18d4f7d931074b81e" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
W20240501 14:00:50.795948 4969 leader_election.cc:341] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.798894 7639 ts_tablet_manager.cc:1367] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20240501 14:00:50.800963 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9921ad9ce549e18d4f7d931074b81e" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:50.801211 5164 raft_consensus.cc:2454] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:50.801637 4967 leader_election.cc:302] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:50.802007 7594 raft_consensus.cc:2781] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:50.802116 7594 raft_consensus.cc:482] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:50.802204 7594 raft_consensus.cc:3037] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.805877 7594 raft_consensus.cc:504] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.807548 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9921ad9ce549e18d4f7d931074b81e" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
W20240501 14:00:50.808403 4969 leader_election.cc:341] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.812234 7594 leader_election.cc:288] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:50.812536 7633 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:50.812722 7633 ts_tablet_manager.cc:1367] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.018s
I20240501 14:00:50.813026 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9921ad9ce549e18d4f7d931074b81e" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:50.813725 5164 raft_consensus.cc:3037] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.813997 7633 raft_consensus.cc:348] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.814213 7633 raft_consensus.cc:374] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.814275 7633 raft_consensus.cc:729] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.814541 7633 consensus_queue.cc:260] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.815086 7633 ts_tablet_manager.cc:1398] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:50.813974 7639 raft_consensus.cc:348] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:50.815212 7639 raft_consensus.cc:374] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.815310 7639 raft_consensus.cc:729] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.816277 7639 consensus_queue.cc:260] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:50.816756 5164 raft_consensus.cc:2454] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:50.821066 4967 leader_election.cc:302] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:50.821511 7594 raft_consensus.cc:2781] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:50.821681 7594 raft_consensus.cc:686] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:50.822012 7594 consensus_queue.cc:237] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.825969 7639 ts_tablet_manager.cc:1398] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:00:50.827549 7639 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:50.829370 7639 tablet_bootstrap.cc:654] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.831324 7655 ts_tablet_manager.cc:902] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:50.833204 1999 catalog_manager.cc:5484] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:50.835453 7641 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:50.835623 7641 ts_tablet_manager.cc:1367] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.003s
I20240501 14:00:50.840034 7641 raft_consensus.cc:348] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.840260 7641 raft_consensus.cc:374] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.840359 7641 raft_consensus.cc:729] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.843576 7655 tablet_copy_client.cc:308] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:50.843834 7641 consensus_queue.cc:260] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.844308 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet f67ca7eac95d4a7fbd28667bdde44a22 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:41439)
I20240501 14:00:50.844449 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet f67ca7eac95d4a7fbd28667bdde44a22 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:41439: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-f67ca7eac95d4a7fbd28667bdde44a22
I20240501 14:00:50.846833 7641 ts_tablet_manager.cc:1398] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240501 14:00:50.846995 2428 tablet_copy_source_session.cc:205] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:50.847097 7639 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:50.847139 7641 tablet_bootstrap.cc:492] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:50.847257 7639 ts_tablet_manager.cc:1367] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240501 14:00:50.848035 7639 raft_consensus.cc:348] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.848371 7639 raft_consensus.cc:374] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.848470 7639 raft_consensus.cc:729] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.848721 7639 consensus_queue.cc:260] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.848985 7641 tablet_bootstrap.cc:654] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.849550 7639 ts_tablet_manager.cc:1398] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:50.859504 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67ca7eac95d4a7fbd28667bdde44a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.865144 7655 tablet_copy_client.cc:783] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:50.865494 7641 tablet_bootstrap.cc:492] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:50.865550 7655 tablet_copy_client.cc:647] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:50.865759 7641 ts_tablet_manager.cc:1367] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.001s
I20240501 14:00:50.866822 7641 raft_consensus.cc:348] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.867110 7641 raft_consensus.cc:374] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.867229 7641 raft_consensus.cc:729] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.867578 7641 consensus_queue.cc:260] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.869644 7641 ts_tablet_manager.cc:1398] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:50.869982 7641 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:50.871438 7638 raft_consensus.cc:482] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.871596 7638 raft_consensus.cc:504] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.872299 7638 leader_election.cc:288] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:50.873039 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878eb076eee94aa8988b3588d5395cba" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:50.873255 5293 raft_consensus.cc:2454] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:50.874213 7641 tablet_bootstrap.cc:654] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:50.874900 2213 leader_election.cc:302] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:50.875418 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878eb076eee94aa8988b3588d5395cba" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:50.875681 5164 raft_consensus.cc:2454] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:50.878803 7638 raft_consensus.cc:2781] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:50.878945 7638 raft_consensus.cc:482] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:50.879035 7638 raft_consensus.cc:3037] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.898085 7638 raft_consensus.cc:504] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.900307 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878eb076eee94aa8988b3588d5395cba" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:50.900620 5164 raft_consensus.cc:3037] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.906848 5164 raft_consensus.cc:2454] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:50.907939 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878eb076eee94aa8988b3588d5395cba" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:50.908210 5293 raft_consensus.cc:3037] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.911484 5293 raft_consensus.cc:2454] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:50.912324 2213 leader_election.cc:302] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:50.912703 6764 raft_consensus.cc:2781] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:50.913276 6764 raft_consensus.cc:686] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:50.913590 6764 consensus_queue.cc:237] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:50.916561 7638 leader_election.cc:288] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:50.921777 7655 tablet_copy_client.cc:523] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:50.935997 7363 raft_consensus.cc:482] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:50.936223 7655 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:50.936182 7363 raft_consensus.cc:504] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.936882 7363 leader_election.cc:288] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:50.938063 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f03024767442ac911a8a44f7ecdceb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:50.938097 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f03024767442ac911a8a44f7ecdceb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:50.938318 5031 raft_consensus.cc:2454] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:50.938796 5099 leader_election.cc:302] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:50.940716 7159 raft_consensus.cc:2781] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:50.940842 7159 raft_consensus.cc:482] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:50.940939 7159 raft_consensus.cc:3037] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:50.943013 5099 leader_election.cc:341] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.946302 7159 raft_consensus.cc:504] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.948880 7159 leader_election.cc:288] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:50.949741 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f03024767442ac911a8a44f7ecdceb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
W20240501 14:00:50.950963 5099 leader_election.cc:341] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:50.951320 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f03024767442ac911a8a44f7ecdceb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:50.951567 5031 raft_consensus.cc:3037] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.954739 5031 raft_consensus.cc:2454] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:50.955420 5099 leader_election.cc:302] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:00:50.956013 7159 raft_consensus.cc:2781] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:50.956204 7159 raft_consensus.cc:686] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:50.956563 7159 consensus_queue.cc:237] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.981601 2000 catalog_manager.cc:5484] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:50.984470 7641 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:50.984894 7641 ts_tablet_manager.cc:1367] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.115s user 0.049s sys 0.045s
I20240501 14:00:50.985911 7641 raft_consensus.cc:348] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.986258 7641 raft_consensus.cc:374] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:50.986727 1999 catalog_manager.cc:5484] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:50.987257 7641 raft_consensus.cc:729] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:50.987560 7641 consensus_queue.cc:260] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:50.990649 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:50.992480 7641 ts_tablet_manager.cc:1398] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:50.997257 5293 raft_consensus.cc:3037] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:50.997435 5164 raft_consensus.cc:1261] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:50.998383 7646 consensus_queue.cc:1035] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 14:00:51.000624 5293 raft_consensus.cc:1261] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.002333 7594 consensus_queue.cc:1035] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.014532 7655 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:51.015024 7655 tablet_bootstrap.cc:492] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:51.015349 7655 ts_tablet_manager.cc:1367] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.079s user 0.043s sys 0.013s
I20240501 14:00:51.016191 7655 raft_consensus.cc:348] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:51.016500 7655 raft_consensus.cc:729] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:51.016726 7655 consensus_queue.cc:260] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:51.018795 7655 ts_tablet_manager.cc:1398] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:51.019690 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-f67ca7eac95d4a7fbd28667bdde44a22 received from {username='slave'} at 127.28.17.6:41439
I20240501 14:00:51.019810 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-f67ca7eac95d4a7fbd28667bdde44a22 on tablet f67ca7eac95d4a7fbd28667bdde44a22 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:51.027899 1999 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet bb9921ad9ce549e18d4f7d931074b81e (table rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885]). Expected version 1 got 0
I20240501 14:00:51.037429 7687 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:51.038286 7686 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:51.043313 1997 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet bb9921ad9ce549e18d4f7d931074b81e (table rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885]). Expected version 1 got 0
I20240501 14:00:51.045656 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885] alter complete (version 1)
I20240501 14:00:51.046154 2000 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet bb9921ad9ce549e18d4f7d931074b81e (table rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885]). Expected version 1 got 0
I20240501 14:00:51.046708 7686 tablet.cc:1620] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:51.048162 7687 tablet.cc:1620] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:51.053041 6764 raft_consensus.cc:482] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.053282 6764 raft_consensus.cc:504] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:51.054044 6764 leader_election.cc:288] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:51.055621 7689 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:51.055838 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe1b54933ad446082b1d2840b2c6e49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:51.056114 2146 raft_consensus.cc:2454] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:51.056337 7689 tablet.cc:1620] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:51.058730 2211 leader_election.cc:302] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:51.059172 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe1b54933ad446082b1d2840b2c6e49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:51.059469 5164 raft_consensus.cc:2454] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:51.059605 1999 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet bb9921ad9ce549e18d4f7d931074b81e (table rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885]). Expected version 1 got 0
I20240501 14:00:51.060631 6764 raft_consensus.cc:2781] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.060765 6764 raft_consensus.cc:482] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.060851 6764 raft_consensus.cc:3037] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.063427 6764 raft_consensus.cc:504] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:51.065127 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe1b54933ad446082b1d2840b2c6e49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:51.065433 5164 raft_consensus.cc:3037] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.071297 5164 raft_consensus.cc:2454] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:51.071843 2211 leader_election.cc:302] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:51.072142 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe1b54933ad446082b1d2840b2c6e49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:51.072252 7638 raft_consensus.cc:2781] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.072392 7638 raft_consensus.cc:686] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:51.072376 2143 raft_consensus.cc:3037] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.072678 7638 consensus_queue.cc:237] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:51.071879 6764 leader_election.cc:288] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:51.075102 2143 raft_consensus.cc:2454] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:51.077126 1997 catalog_manager.cc:5484] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:51.091497 5164 consensus_queue.cc:237] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
W20240501 14:00:51.094440 5099 consensus_peers.cc:485] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83392c08bf314fcd98833335c247ccb1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:51.094998 2537 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.095227 4899 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.095820 7363 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:51.096700 7159 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:51.099555 7695 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:51.100051 2537 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:51.105711 4899 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:51.106369 1997 catalog_manager.cc:5484] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:51.108206 7214 raft_consensus.cc:482] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.108408 7214 raft_consensus.cc:504] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:51.109259 7214 leader_election.cc:288] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:51.110296 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f03024767442ac911a8a44f7ecdceb" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:51.110328 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f03024767442ac911a8a44f7ecdceb" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:51.110589 5031 raft_consensus.cc:2379] T 36f03024767442ac911a8a44f7ecdceb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a4b9a96e9734baa8a03d6056bfa934c in current term 1: Already voted for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in this term.
I20240501 14:00:51.111444 5230 leader_election.cc:302] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:51.112322 7214 raft_consensus.cc:3037] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.117087 7214 raft_consensus.cc:2726] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:51.150710 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:51.158175 7686 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:51.158757 7686 tablet.cc:1620] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:51.159224 7687 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:51.159670 7687 tablet.cc:1620] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:51.161547 7689 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:51.162117 7689 tablet.cc:1620] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:51.163780 5031 raft_consensus.cc:1203] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:51.172183 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=67669095d6c64cc2874b3d68d3751885] alter complete (version 2)
I20240501 14:00:51.276995 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:51.284543 7686 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:51.285495 7687 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:51.285382 7689 tablet.cc:1722] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:51.294633 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=67669095d6c64cc2874b3d68d3751885] alter complete (version 3)
I20240501 14:00:51.299475 6764 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.305737 7638 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.336500 7363 consensus_queue.cc:1035] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.341195 7695 consensus_queue.cc:1035] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.398943 1997 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:51.399159 1997 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:51.404024 1997 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet bb9921ad9ce549e18d4f7d931074b81e
I20240501 14:00:51.405050 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9921ad9ce549e18d4f7d931074b81e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:51 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:51.405246 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9921ad9ce549e18d4f7d931074b81e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:51 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:51.405457 5012 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9921ad9ce549e18d4f7d931074b81e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:51 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:51.417601 7725 tablet_replica.cc:330] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:51.418104 7725 raft_consensus.cc:2229] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:51.418668 7725 raft_consensus.cc:2258] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:51.420123 7725 ts_tablet_manager.cc:1875] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:51.420470 7723 tablet_replica.cc:330] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:51.420848 7723 raft_consensus.cc:2229] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:51.421159 7723 raft_consensus.cc:2258] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:51.423192 7723 ts_tablet_manager.cc:1875] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:51.432433 7724 tablet_replica.cc:330] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:51.432639 7723 ts_tablet_manager.cc:1888] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:51.432745 7724 raft_consensus.cc:2229] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:51.432853 7723 log.cc:1196] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/bb9921ad9ce549e18d4f7d931074b81e
I20240501 14:00:51.433338 7723 ts_tablet_manager.cc:1909] T bb9921ad9ce549e18d4f7d931074b81e P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:51.434590 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet bb9921ad9ce549e18d4f7d931074b81e (table rebalancer_dynamic_table_new_name [id=67669095d6c64cc2874b3d68d3751885]) successfully deleted
I20240501 14:00:51.433099 7724 raft_consensus.cc:2258] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:51.435731 7724 ts_tablet_manager.cc:1875] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:51.441885 7724 ts_tablet_manager.cc:1888] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:51.442025 7724 log.cc:1196] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/bb9921ad9ce549e18d4f7d931074b81e
I20240501 14:00:51.446718 7725 ts_tablet_manager.cc:1888] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:51.446869 7725 log.cc:1196] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/bb9921ad9ce549e18d4f7d931074b81e
I20240501 14:00:51.447305 7725 ts_tablet_manager.cc:1909] T bb9921ad9ce549e18d4f7d931074b81e P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:51.449721 7724 ts_tablet_manager.cc:1909] T bb9921ad9ce549e18d4f7d931074b81e P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:51.449683 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet bb9921ad9ce549e18d4f7d931074b81e (table rebalancer_dynamic_table_new_name [id=67669095d6c64cc2874b3d68d3751885]) successfully deleted
I20240501 14:00:51.450814 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet bb9921ad9ce549e18d4f7d931074b81e (table rebalancer_dynamic_table_new_name [id=67669095d6c64cc2874b3d68d3751885]) successfully deleted
I20240501 14:00:51.507751 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:51.517763 2387 tablet_service.cc:1467] Processing CreateTablet for tablet f8613a81e9a042ecbfe5dedd828a824e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1]), partition=
I20240501 14:00:51.518177 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8613a81e9a042ecbfe5dedd828a824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.518167 5276 tablet_service.cc:1467] Processing CreateTablet for tablet f8613a81e9a042ecbfe5dedd828a824e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1]), partition=
I20240501 14:00:51.518613 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8613a81e9a042ecbfe5dedd828a824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.522709 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f8613a81e9a042ecbfe5dedd828a824e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1]), partition=
I20240501 14:00:51.523084 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8613a81e9a042ecbfe5dedd828a824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.530351 7728 tablet_bootstrap.cc:492] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:51.532166 7728 tablet_bootstrap.cc:654] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.536530 7731 tablet_bootstrap.cc:492] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:51.539958 7731 tablet_bootstrap.cc:654] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.543500 7730 tablet_bootstrap.cc:492] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:51.545295 7730 tablet_bootstrap.cc:654] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.552316 7638 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.557225 7728 tablet_bootstrap.cc:492] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:51.557417 7728 ts_tablet_manager.cc:1367] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.009s
I20240501 14:00:51.558405 7728 raft_consensus.cc:348] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.558681 7728 raft_consensus.cc:374] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.559438 7728 raft_consensus.cc:729] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.559698 7728 consensus_queue.cc:260] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.562340 7728 ts_tablet_manager.cc:1398] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:51.580231 6764 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.592098 7730 tablet_bootstrap.cc:492] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:51.593001 7731 tablet_bootstrap.cc:492] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:51.593163 7731 ts_tablet_manager.cc:1367] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.005s
I20240501 14:00:51.593856 7730 ts_tablet_manager.cc:1367] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.015s
I20240501 14:00:51.594849 7730 raft_consensus.cc:348] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.595125 7730 raft_consensus.cc:374] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.595221 7730 raft_consensus.cc:729] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.595504 7730 consensus_queue.cc:260] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.596158 7730 ts_tablet_manager.cc:1398] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:51.598080 7731 raft_consensus.cc:348] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.598390 7731 raft_consensus.cc:374] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.598557 7731 raft_consensus.cc:729] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.598912 7731 consensus_queue.cc:260] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.600754 7731 ts_tablet_manager.cc:1398] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:51.635838 7214 raft_consensus.cc:482] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.636011 7214 raft_consensus.cc:504] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.636703 7214 leader_election.cc:288] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:51.637193 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8613a81e9a042ecbfe5dedd828a824e" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:51.637415 2406 raft_consensus.cc:2454] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:51.637943 5228 leader_election.cc:302] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:51.638258 7214 raft_consensus.cc:2781] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.638386 7214 raft_consensus.cc:482] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.638484 7214 raft_consensus.cc:3037] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.638993 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8613a81e9a042ecbfe5dedd828a824e" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:51.639205 2277 raft_consensus.cc:2454] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:51.641667 7214 raft_consensus.cc:504] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.643182 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8613a81e9a042ecbfe5dedd828a824e" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:51.643296 7214 leader_election.cc:288] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:51.643380 2406 raft_consensus.cc:3037] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.644634 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8613a81e9a042ecbfe5dedd828a824e" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:51.644860 2277 raft_consensus.cc:3037] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.646287 2406 raft_consensus.cc:2454] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:51.648341 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:51.652065 5228 leader_election.cc:302] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:51.653905 2277 raft_consensus.cc:2454] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:51.659511 7738 ts_tablet_manager.cc:902] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:51.659504 7214 raft_consensus.cc:2781] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.661710 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 9eec5cc39eda4f6d85f83959952a322c (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.662222 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eec5cc39eda4f6d85f83959952a322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.663537 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 9eec5cc39eda4f6d85f83959952a322c (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.663954 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eec5cc39eda4f6d85f83959952a322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.666956 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 9eec5cc39eda4f6d85f83959952a322c (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.668082 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eec5cc39eda4f6d85f83959952a322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.668059 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc3084368de4add8809f10451d96894 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.668442 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc3084368de4add8809f10451d96894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.669889 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc3084368de4add8809f10451d96894 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.670300 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc3084368de4add8809f10451d96894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.673549 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8dfee2d66f44ba9ebcf74f6e570230 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.673882 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8dfee2d66f44ba9ebcf74f6e570230 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.674019 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8dfee2d66f44ba9ebcf74f6e570230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.674111 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc3084368de4add8809f10451d96894 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.674293 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8dfee2d66f44ba9ebcf74f6e570230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.674471 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc3084368de4add8809f10451d96894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.679034 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 6d8dfee2d66f44ba9ebcf74f6e570230 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:51.679451 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d8dfee2d66f44ba9ebcf74f6e570230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.688253 7728 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:51.689164 7730 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:51.689877 7728 tablet_bootstrap.cc:654] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.690474 7730 tablet_bootstrap.cc:654] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.694993 7744 tablet_bootstrap.cc:492] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:51.696902 7744 tablet_bootstrap.cc:654] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.698279 7745 tablet_bootstrap.cc:492] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:51.700312 7745 tablet_bootstrap.cc:654] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.714242 7169 raft_consensus.cc:1050] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:51.715181 7169 consensus_queue.cc:237] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:51.717219 7745 tablet_bootstrap.cc:492] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:51.721537 7745 ts_tablet_manager.cc:1367] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
I20240501 14:00:51.722493 7745 raft_consensus.cc:348] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.722843 7745 raft_consensus.cc:374] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.722985 7745 raft_consensus.cc:729] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.723280 7745 consensus_queue.cc:260] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.727669 7745 ts_tablet_manager.cc:1398] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:51.739912 7743 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:51.744913 7753 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:51.745211 5032 raft_consensus.cc:1261] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:51.745779 7692 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:51.746884 7744 tablet_bootstrap.cc:492] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:51.747105 7753 tablet_bootstrap.cc:654] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.747992 7743 tablet_bootstrap.cc:654] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.748272 7744 ts_tablet_manager.cc:1367] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.011s
I20240501 14:00:51.749752 7744 raft_consensus.cc:348] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.751880 7744 raft_consensus.cc:374] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.752046 7744 raft_consensus.cc:729] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.752372 7744 consensus_queue.cc:260] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.753093 2143 raft_consensus.cc:1261] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:51.755209 7728 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:51.755769 7692 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:51.755985 7214 raft_consensus.cc:686] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:51.756676 7753 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:51.757030 2537 raft_consensus.cc:1261] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:51.757354 7753 ts_tablet_manager.cc:1367] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:00:51.757979 7692 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:51.758200 7753 raft_consensus.cc:348] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.758388 7753 raft_consensus.cc:374] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.756280 7214 consensus_queue.cc:237] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.763279 7744 ts_tablet_manager.cc:1398] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:00:51.758473 7753 raft_consensus.cc:729] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.764957 7753 consensus_queue.cc:260] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.767938 7753 ts_tablet_manager.cc:1398] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:00:51.770721 7743 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:51.770987 7743 ts_tablet_manager.cc:1367] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240501 14:00:51.771703 7743 raft_consensus.cc:348] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.772076 7743 raft_consensus.cc:374] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.772176 7743 raft_consensus.cc:729] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.772476 7743 consensus_queue.cc:260] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.780843 7730 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:51.784123 7728 ts_tablet_manager.cc:1367] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.096s user 0.007s sys 0.002s
I20240501 14:00:51.784996 7728 raft_consensus.cc:348] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.785580 7730 ts_tablet_manager.cc:1367] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.096s user 0.000s sys 0.010s
I20240501 14:00:51.786284 7730 raft_consensus.cc:348] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.786417 7728 raft_consensus.cc:374] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.786532 7730 raft_consensus.cc:374] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.786612 7730 raft_consensus.cc:729] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.786864 7730 consensus_queue.cc:260] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.787765 7743 ts_tablet_manager.cc:1398] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240501 14:00:51.789153 7730 ts_tablet_manager.cc:1398] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:51.790205 2002 catalog_manager.cc:5484] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:51.793252 7728 raft_consensus.cc:729] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.793584 7728 consensus_queue.cc:260] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.793972 7743 tablet_bootstrap.cc:492] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:51.794718 7728 ts_tablet_manager.cc:1398] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 14:00:51.795075 7728 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:51.795593 7743 tablet_bootstrap.cc:654] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.796454 7279 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:51.797009 7728 tablet_bootstrap.cc:654] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.802592 7738 tablet_copy_client.cc:308] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:00:51.802793 2537 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:51.803666 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 83392c08bf314fcd98833335c247ccb1 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:54429)
I20240501 14:00:51.803820 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 83392c08bf314fcd98833335c247ccb1 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:54429: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-83392c08bf314fcd98833335c247ccb1
I20240501 14:00:51.807404 7743 tablet_bootstrap.cc:492] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:51.807714 7743 ts_tablet_manager.cc:1367] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240501 14:00:51.808526 7743 raft_consensus.cc:348] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.808831 7743 raft_consensus.cc:374] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.808945 7743 raft_consensus.cc:729] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.809295 7743 consensus_queue.cc:260] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.806700 5184 tablet_copy_source_session.cc:205] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:51.813465 7738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83392c08bf314fcd98833335c247ccb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:51.818102 2143 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:51.818703 5032 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:51.821565 1997 catalog_manager.cc:5484] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:51.823297 7743 ts_tablet_manager.cc:1398] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 14:00:51.823724 7743 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:51.827642 7743 tablet_bootstrap.cc:654] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:51.829093 7738 tablet_copy_client.cc:783] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:51.829386 7738 tablet_copy_client.cc:647] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:51.834270 7728 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:51.834455 7728 ts_tablet_manager.cc:1367] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.010s
I20240501 14:00:51.835325 7743 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:51.835250 7728 raft_consensus.cc:348] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.835490 7728 raft_consensus.cc:374] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.835582 7728 raft_consensus.cc:729] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.835614 7743 ts_tablet_manager.cc:1367] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240501 14:00:51.835924 7728 consensus_queue.cc:260] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.836165 7751 raft_consensus.cc:482] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.836205 7738 tablet_copy_client.cc:523] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:51.836284 7751 raft_consensus.cc:504] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.837077 7751 leader_election.cc:288] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:51.836992 7743 raft_consensus.cc:348] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.837502 7743 raft_consensus.cc:374] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:51.837611 7743 raft_consensus.cc:729] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:51.838055 7743 consensus_queue.cc:260] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.838312 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:51.838584 4899 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:51.838886 7743 ts_tablet_manager.cc:1398] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:51.839051 5229 leader_election.cc:302] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:51.839133 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:51.839329 2537 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:51.839370 7751 raft_consensus.cc:2781] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.839494 7751 raft_consensus.cc:482] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.839586 7751 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.841547 7738 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:51.842069 7751 raft_consensus.cc:504] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.843310 2406 consensus_queue.cc:237] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:51.844133 7751 leader_election.cc:288] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:51.844622 7728 ts_tablet_manager.cc:1398] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:00:51.845921 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:51.846128 4899 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.847815 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:51.848026 2537 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.849402 4899 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:51.849913 5229 leader_election.cc:302] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:51.852094 2537 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:51.854771 2143 raft_consensus.cc:1261] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:51.855369 7751 raft_consensus.cc:2781] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.855325 7692 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:51.855525 7751 raft_consensus.cc:686] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:51.855782 7751 consensus_queue.cc:237] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.858892 7738 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:51.859427 7738 tablet_bootstrap.cc:492] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:51.859797 7738 ts_tablet_manager.cc:1367] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20240501 14:00:51.860970 7738 raft_consensus.cc:348] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:51.861452 7738 raft_consensus.cc:729] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:51.861752 7738 consensus_queue.cc:260] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:51.862030 5033 raft_consensus.cc:1261] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:51.862587 7738 ts_tablet_manager.cc:1398] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:51.862883 7692 consensus_queue.cc:1035] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:51.870060 7169 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:51.871610 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-83392c08bf314fcd98833335c247ccb1 received from {username='slave'} at 127.28.17.6:54429
I20240501 14:00:51.871755 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-83392c08bf314fcd98833335c247ccb1 on tablet 83392c08bf314fcd98833335c247ccb1 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:51.881326 7708 raft_consensus.cc:482] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.881525 7708 raft_consensus.cc:504] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.882205 7708 leader_election.cc:288] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:51.883638 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8dfee2d66f44ba9ebcf74f6e570230" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:51.883862 2143 raft_consensus.cc:2454] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:51.884738 2471 leader_election.cc:302] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:51.885610 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8dfee2d66f44ba9ebcf74f6e570230" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:51.885815 5033 raft_consensus.cc:2454] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:51.886556 7388 raft_consensus.cc:2781] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.886631 7388 raft_consensus.cc:482] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.886680 7388 raft_consensus.cc:3037] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.889185 7388 raft_consensus.cc:504] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.889796 7388 leader_election.cc:288] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:51.892673 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8dfee2d66f44ba9ebcf74f6e570230" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:51.892956 5032 raft_consensus.cc:3037] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.897850 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d8dfee2d66f44ba9ebcf74f6e570230" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:51.898089 2143 raft_consensus.cc:3037] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.900825 2143 raft_consensus.cc:2454] T 6d8dfee2d66f44ba9ebcf74f6e570230 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:51.900650 1997 catalog_manager.cc:5484] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:51.902071 2471 leader_election.cc:302] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:51.902388 7708 raft_consensus.cc:2781] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.902549 7708 raft_consensus.cc:686] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:51.902791 7708 consensus_queue.cc:237] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:51.904115 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f67ca7eac95d4a7fbd28667bdde44a22 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:51.904040 2146 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:51.911302 2002 catalog_manager.cc:5484] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:51.912168 1997 catalog_manager.cc:5484] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:51.916577 5033 raft_consensus.cc:2932] T f67ca7eac95d4a7fbd28667bdde44a22 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:51.923935 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet f67ca7eac95d4a7fbd28667bdde44a22 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:51.926488 7777 tablet_replica.cc:330] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:51.926558 5032 raft_consensus.cc:2454] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:51.928551 7777 raft_consensus.cc:2229] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:51.928855 7777 raft_consensus.cc:2258] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:51.929684 7777 ts_tablet_manager.cc:1875] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:51.936797 7751 raft_consensus.cc:482] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:51.937002 7751 raft_consensus.cc:504] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.937376 7777 ts_tablet_manager.cc:1888] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:51.937476 7777 log.cc:1196] T f67ca7eac95d4a7fbd28667bdde44a22 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/f67ca7eac95d4a7fbd28667bdde44a22
I20240501 14:00:51.937664 7751 leader_election.cc:288] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:51.938117 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc3084368de4add8809f10451d96894" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:51.938302 2537 raft_consensus.cc:2454] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:51.938726 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc3084368de4add8809f10451d96894" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:51.938865 5228 leader_election.cc:302] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:51.938925 2406 raft_consensus.cc:2454] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:00:51.939201 7751 raft_consensus.cc:2781] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:51.939285 7751 raft_consensus.cc:482] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:51.939383 7751 raft_consensus.cc:3037] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.942253 7751 raft_consensus.cc:504] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.945233 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc3084368de4add8809f10451d96894" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:51.945467 2537 raft_consensus.cc:3037] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.949304 2537 raft_consensus.cc:2454] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:51.949973 5228 leader_election.cc:302] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:51.950322 7214 raft_consensus.cc:2781] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:51.950486 7214 raft_consensus.cc:686] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:00:51.950847 7214 consensus_queue.cc:237] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:51.952546 7751 leader_election.cc:288] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:51.959380 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet f67ca7eac95d4a7fbd28667bdde44a22 (table rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]) successfully deleted
I20240501 14:00:51.961199 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc3084368de4add8809f10451d96894" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:51.961408 2406 raft_consensus.cc:3037] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:51.961988 1997 catalog_manager.cc:5484] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:51.964334 2406 raft_consensus.cc:2454] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:00:51.974642 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:51.981303 2408 raft_consensus.cc:1261] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.981983 2277 raft_consensus.cc:1261] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:51.981871 7214 consensus_queue.cc:1035] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:51.982851 7214 consensus_queue.cc:1035] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.005048 2002 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet f8613a81e9a042ecbfe5dedd828a824e (table rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1]). Expected version 1 got 0
I20240501 14:00:52.006768 2005 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet f8613a81e9a042ecbfe5dedd828a824e (table rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1]). Expected version 1 got 0
I20240501 14:00:52.014266 7794 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.017203 2002 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet f8613a81e9a042ecbfe5dedd828a824e (table rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1]). Expected version 1 got 0
I20240501 14:00:52.017565 7794 tablet.cc:1620] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:52.016188 7793 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.019112 7793 tablet.cc:1620] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:52.026237 2005 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet f8613a81e9a042ecbfe5dedd828a824e (table rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1]). Expected version 1 got 0
I20240501 14:00:52.027348 7790 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.027945 7790 tablet.cc:1620] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:52.033564 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1] alter complete (version 1)
I20240501 14:00:52.035974 2537 consensus_queue.cc:237] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:52.039845 2406 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:52.040035 2277 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:52.040462 7708 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:52.040724 7388 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:00:52.042100 2473 consensus_peers.cc:485] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d42a56dd51fe410cb42dd284ecbf2748. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:52.044219 7796 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:52.044764 2277 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:52.046269 2408 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:52.050536 2002 catalog_manager.cc:5484] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:52.083175 5032 raft_consensus.cc:1203] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:00:52.137107 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:52.146907 7790 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:52.147497 7790 tablet.cc:1620] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:52.148638 7793 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:52.149106 7793 tablet.cc:1620] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:52.153702 7794 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:52.154156 7794 tablet.cc:1620] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:52.161943 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=4893fe663edf4ef289e0386799ed23a1] alter complete (version 2)
I20240501 14:00:52.266180 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:52.275617 7790 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:52.276171 7794 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:52.277169 7793 tablet.cc:1722] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:52.289558 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=4893fe663edf4ef289e0386799ed23a1] alter complete (version 3)
I20240501 14:00:52.303866 7388 consensus_queue.cc:1035] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.309671 7796 consensus_queue.cc:1035] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.355068 7214 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.360854 7214 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.393939 1999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:52.394196 1999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:52.397215 7746 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.397555 1999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet f8613a81e9a042ecbfe5dedd828a824e
I20240501 14:00:52.398321 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet f8613a81e9a042ecbfe5dedd828a824e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:52 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:52.398764 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet f8613a81e9a042ecbfe5dedd828a824e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:52 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:52.399245 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet f8613a81e9a042ecbfe5dedd828a824e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:52 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:52.400576 7829 tablet_replica.cc:330] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:52.400856 7829 raft_consensus.cc:2229] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:52.401265 7829 raft_consensus.cc:2258] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:52.402887 7828 tablet_replica.cc:330] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:52.403190 7828 raft_consensus.cc:2229] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:52.403455 7828 raft_consensus.cc:2258] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:52.404381 7828 ts_tablet_manager.cc:1875] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:52.404700 7829 ts_tablet_manager.cc:1875] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:52.411134 7828 ts_tablet_manager.cc:1888] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:52.411957 7828 log.cc:1196] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/f8613a81e9a042ecbfe5dedd828a824e
I20240501 14:00:52.412412 7828 ts_tablet_manager.cc:1909] T f8613a81e9a042ecbfe5dedd828a824e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:52.413188 7740 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.413599 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet f8613a81e9a042ecbfe5dedd828a824e (table rebalancer_dynamic_table_new_name [id=4893fe663edf4ef289e0386799ed23a1]) successfully deleted
I20240501 14:00:52.416044 7829 ts_tablet_manager.cc:1888] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:52.416199 7829 log.cc:1196] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/f8613a81e9a042ecbfe5dedd828a824e
I20240501 14:00:52.416738 7829 ts_tablet_manager.cc:1909] T f8613a81e9a042ecbfe5dedd828a824e P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:52.418476 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet f8613a81e9a042ecbfe5dedd828a824e (table rebalancer_dynamic_table_new_name [id=4893fe663edf4ef289e0386799ed23a1]) successfully deleted
I20240501 14:00:52.423141 7827 tablet_replica.cc:330] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:52.423358 7827 raft_consensus.cc:2229] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:52.423552 7827 raft_consensus.cc:2258] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:52.423954 7833 ts_tablet_manager.cc:902] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:52.424517 7827 ts_tablet_manager.cc:1875] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:52.425845 7833 tablet_copy_client.cc:308] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:00:52.426695 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet d42a56dd51fe410cb42dd284ecbf2748 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:45411)
I20240501 14:00:52.426880 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet d42a56dd51fe410cb42dd284ecbf2748 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:45411: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-d42a56dd51fe410cb42dd284ecbf2748
I20240501 14:00:52.428997 2558 tablet_copy_source_session.cc:205] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:52.430199 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42a56dd51fe410cb42dd284ecbf2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.430426 7827 ts_tablet_manager.cc:1888] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:52.430544 7827 log.cc:1196] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8613a81e9a042ecbfe5dedd828a824e
I20240501 14:00:52.430876 7827 ts_tablet_manager.cc:1909] T f8613a81e9a042ecbfe5dedd828a824e P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:52.431732 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet f8613a81e9a042ecbfe5dedd828a824e (table rebalancer_dynamic_table_new_name [id=4893fe663edf4ef289e0386799ed23a1]) successfully deleted
I20240501 14:00:52.434486 7833 tablet_copy_client.cc:783] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:52.434744 7833 tablet_copy_client.cc:647] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:52.436288 7833 tablet_copy_client.cc:523] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:52.438637 7833 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:52.475343 7833 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:52.475826 7833 tablet_bootstrap.cc:492] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:52.476111 7833 ts_tablet_manager.cc:1367] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.038s user 0.037s sys 0.000s
I20240501 14:00:52.476856 7833 raft_consensus.cc:348] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:52.477125 7833 raft_consensus.cc:729] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:52.477342 7833 consensus_queue.cc:260] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:52.477936 7833 ts_tablet_manager.cc:1398] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:52.478734 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-d42a56dd51fe410cb42dd284ecbf2748 received from {username='slave'} at 127.28.17.6:45411
I20240501 14:00:52.478858 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-d42a56dd51fe410cb42dd284ecbf2748 on tablet d42a56dd51fe410cb42dd284ecbf2748 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:52.505628 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:52.514917 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:52.515450 2518 tablet_service.cc:1467] Processing CreateTablet for tablet ee394579959049c4aaadccec3bcd0503 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0]), partition=
I20240501 14:00:52.515622 5143 tablet_service.cc:1467] Processing CreateTablet for tablet ee394579959049c4aaadccec3bcd0503 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0]), partition=
I20240501 14:00:52.515832 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee394579959049c4aaadccec3bcd0503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.515910 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee394579959049c4aaadccec3bcd0503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.521636 5012 tablet_service.cc:1467] Processing CreateTablet for tablet ee394579959049c4aaadccec3bcd0503 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0]), partition=
I20240501 14:00:52.522006 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee394579959049c4aaadccec3bcd0503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.522346 7844 tablet_bootstrap.cc:492] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:52.523612 7844 tablet_bootstrap.cc:654] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.524977 7844 tablet_bootstrap.cc:492] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:52.525125 7844 ts_tablet_manager.cc:1367] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:52.525671 7844 raft_consensus.cc:348] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.525851 7844 raft_consensus.cc:374] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.525926 7844 raft_consensus.cc:729] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.526139 7844 consensus_queue.cc:260] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.526625 7844 ts_tablet_manager.cc:1398] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 14:00:52.536012 7848 tablet_bootstrap.cc:492] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:52.542112 7849 tablet_bootstrap.cc:492] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:52.543983 7849 tablet_bootstrap.cc:654] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.544840 7848 tablet_bootstrap.cc:654] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.548426 5014 tablet_service.cc:1467] Processing CreateTablet for tablet 8e605c78442442dd82c598bac2fd84cd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.548439 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 0a489f3c24c54297a0569dd560cf159e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.548872 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e605c78442442dd82c598bac2fd84cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.548888 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a489f3c24c54297a0569dd560cf159e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.549638 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 0a489f3c24c54297a0569dd560cf159e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.549849 5144 tablet_service.cc:1467] Processing CreateTablet for tablet a095254d78344a28ba5487f742bcfb4d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.550053 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a489f3c24c54297a0569dd560cf159e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.550204 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 8e605c78442442dd82c598bac2fd84cd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.550563 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e605c78442442dd82c598bac2fd84cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.551183 2387 tablet_service.cc:1467] Processing CreateTablet for tablet a095254d78344a28ba5487f742bcfb4d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.551595 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a095254d78344a28ba5487f742bcfb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.554932 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 8e605c78442442dd82c598bac2fd84cd (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.555387 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e605c78442442dd82c598bac2fd84cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.556010 4879 tablet_service.cc:1467] Processing CreateTablet for tablet a095254d78344a28ba5487f742bcfb4d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:52.556438 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a095254d78344a28ba5487f742bcfb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.556775 7796 raft_consensus.cc:482] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.556918 7796 raft_consensus.cc:504] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.557709 7796 leader_election.cc:288] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:52.559819 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a095254d78344a28ba5487f742bcfb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.562920 7853 tablet_bootstrap.cc:492] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:52.564412 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee394579959049c4aaadccec3bcd0503" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:52.565181 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee394579959049c4aaadccec3bcd0503" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:52.565521 7853 tablet_bootstrap.cc:654] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:52.565841 2473 leader_election.cc:341] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:52.568375 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 0a489f3c24c54297a0569dd560cf159e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
W20240501 14:00:52.569324 2471 leader_election.cc:341] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:52.569502 2471 leader_election.cc:302] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:52.569896 7388 raft_consensus.cc:2726] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:52.570119 7851 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:52.569423 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a489f3c24c54297a0569dd560cf159e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:52.571878 7851 tablet_bootstrap.cc:654] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.572783 7852 tablet_bootstrap.cc:492] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:52.578630 7852 tablet_bootstrap.cc:654] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.585253 7848 tablet_bootstrap.cc:492] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:52.585551 7848 ts_tablet_manager.cc:1367] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.008s
I20240501 14:00:52.586802 7848 raft_consensus.cc:348] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.587111 7848 raft_consensus.cc:374] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.587250 7848 raft_consensus.cc:729] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.587574 7848 consensus_queue.cc:260] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.588910 7852 tablet_bootstrap.cc:492] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:52.589439 7848 ts_tablet_manager.cc:1398] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:52.589756 7848 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:52.590152 7851 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:52.590368 7852 ts_tablet_manager.cc:1367] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240501 14:00:52.590628 7849 tablet_bootstrap.cc:492] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:52.591455 7848 tablet_bootstrap.cc:654] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.591398 7852 raft_consensus.cc:348] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.591769 7849 ts_tablet_manager.cc:1367] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.006s
I20240501 14:00:52.592674 7849 raft_consensus.cc:348] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.593046 7851 ts_tablet_manager.cc:1367] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 14:00:52.593950 7852 raft_consensus.cc:374] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.594039 7852 raft_consensus.cc:729] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.594633 7852 consensus_queue.cc:260] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.595132 7849 raft_consensus.cc:374] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.593858 7851 raft_consensus.cc:348] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.595453 7849 raft_consensus.cc:729] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.595600 7851 raft_consensus.cc:374] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.595963 7853 tablet_bootstrap.cc:492] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:52.596118 7853 ts_tablet_manager.cc:1367] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.012s
I20240501 14:00:52.598127 7851 raft_consensus.cc:729] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.598412 7853 raft_consensus.cc:348] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.598646 7853 raft_consensus.cc:374] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.598733 7853 raft_consensus.cc:729] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.595963 7849 consensus_queue.cc:260] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.597615 7852 ts_tablet_manager.cc:1398] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:52.598981 7853 consensus_queue.cc:260] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.599627 7853 ts_tablet_manager.cc:1398] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:52.601054 7849 ts_tablet_manager.cc:1398] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:00:52.598438 7851 consensus_queue.cc:260] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.601871 7851 ts_tablet_manager.cc:1398] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:00:52.602325 7849 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:52.606748 7279 raft_consensus.cc:482] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.606916 7279 raft_consensus.cc:504] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.607587 7279 leader_election.cc:288] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:52.608311 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a095254d78344a28ba5487f742bcfb4d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:52.608603 7853 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
W20240501 14:00:52.609014 2341 leader_election.cc:341] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:52.610126 7849 tablet_bootstrap.cc:654] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.611169 7848 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:52.611402 7848 ts_tablet_manager.cc:1367] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.007s
I20240501 14:00:52.611730 7853 tablet_bootstrap.cc:654] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.612308 7848 raft_consensus.cc:348] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.612637 7848 raft_consensus.cc:374] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.612764 7848 raft_consensus.cc:729] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.613039 7848 consensus_queue.cc:260] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.614866 7848 ts_tablet_manager.cc:1398] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:52.615190 7848 tablet_bootstrap.cc:492] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:52.615314 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a095254d78344a28ba5487f742bcfb4d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:52.615556 4901 raft_consensus.cc:2454] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:52.616104 2342 leader_election.cc:302] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:52.616540 7279 raft_consensus.cc:2781] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.616634 7279 raft_consensus.cc:482] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.616703 7279 raft_consensus.cc:3037] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.617202 7848 tablet_bootstrap.cc:654] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.619526 7849 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:52.619807 7849 ts_tablet_manager.cc:1367] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.004s
I20240501 14:00:52.620666 7849 raft_consensus.cc:348] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.620980 7849 raft_consensus.cc:374] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.621115 7849 raft_consensus.cc:729] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.621469 7849 consensus_queue.cc:260] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.622143 7849 ts_tablet_manager.cc:1398] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:52.622566 7849 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:52.624349 7849 tablet_bootstrap.cc:654] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.628432 7849 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:52.626669 7279 raft_consensus.cc:504] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.629351 7849 ts_tablet_manager.cc:1367] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:52.630357 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a095254d78344a28ba5487f742bcfb4d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:52.630594 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a095254d78344a28ba5487f742bcfb4d" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:52.630782 4901 raft_consensus.cc:3037] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.630342 7849 raft_consensus.cc:348] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.631963 7849 raft_consensus.cc:374] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.632205 7849 raft_consensus.cc:729] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.632584 7849 consensus_queue.cc:260] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.633333 4901 raft_consensus.cc:2454] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:52.633889 7849 ts_tablet_manager.cc:1398] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240501 14:00:52.634821 2341 leader_election.cc:341] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:52.635330 2342 leader_election.cc:302] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:00:52.636066 7169 raft_consensus.cc:2781] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.636191 7169 raft_consensus.cc:686] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:52.636474 7169 consensus_queue.cc:237] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.644774 7279 leader_election.cc:288] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:52.671324 7848 tablet_bootstrap.cc:492] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:52.671541 7848 ts_tablet_manager.cc:1367] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.024s
I20240501 14:00:52.672380 7848 raft_consensus.cc:348] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.672616 7848 raft_consensus.cc:374] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.672713 7848 raft_consensus.cc:729] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.672987 7848 consensus_queue.cc:260] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
W20240501 14:00:52.681797 2211 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:41309, user_credentials={real_user=slave}} blocked reactor thread for 82322.8us
I20240501 14:00:52.681540 7848 ts_tablet_manager.cc:1398] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240501 14:00:52.683717 7848 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:52.686782 7648 raft_consensus.cc:482] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.686959 7648 raft_consensus.cc:504] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.687695 7648 leader_election.cc:288] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:52.683027 2002 catalog_manager.cc:5484] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:52.689872 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee394579959049c4aaadccec3bcd0503" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:52.690124 5161 raft_consensus.cc:2454] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:52.690969 4967 leader_election.cc:302] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:52.691547 7594 raft_consensus.cc:2781] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.691646 7594 raft_consensus.cc:482] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.691743 7594 raft_consensus.cc:3037] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.691809 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee394579959049c4aaadccec3bcd0503" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:52.692004 2537 raft_consensus.cc:2454] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:52.694322 7594 raft_consensus.cc:504] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.695701 7594 leader_election.cc:288] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:52.697216 7695 raft_consensus.cc:482] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.697366 7695 raft_consensus.cc:504] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:52.698117 7695 leader_election.cc:288] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:52.699141 7848 tablet_bootstrap.cc:654] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:52.699204 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a095254d78344a28ba5487f742bcfb4d" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:52.700091 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a095254d78344a28ba5487f742bcfb4d" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:52.700405 4899 raft_consensus.cc:2379] T a095254d78344a28ba5487f742bcfb4d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9f94ddc5ff64ad79b9f01846a20e2fd in current term 1: Already voted for candidate a64f34785a7b44fa8070b0452a4e0ddd in this term.
I20240501 14:00:52.701054 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee394579959049c4aaadccec3bcd0503" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:52.701256 5161 raft_consensus.cc:3037] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.706048 5098 leader_election.cc:302] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:52.706329 7809 raft_consensus.cc:1050] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:52.707208 7809 consensus_queue.cc:237] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:52.709040 5161 raft_consensus.cc:2454] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:52.709686 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee394579959049c4aaadccec3bcd0503" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:52.709892 2537 raft_consensus.cc:3037] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.712800 2537 raft_consensus.cc:2454] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:52.714995 4967 leader_election.cc:302] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:52.715322 7594 raft_consensus.cc:2781] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.715478 7594 raft_consensus.cc:686] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:52.715960 7594 consensus_queue.cc:237] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:52.709417 7695 raft_consensus.cc:3037] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.719626 7695 raft_consensus.cc:2726] T a095254d78344a28ba5487f742bcfb4d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:52.738663 7853 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:52.739485 7848 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:52.739485 7853 ts_tablet_manager.cc:1367] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.130s user 0.057s sys 0.045s
I20240501 14:00:52.739679 7848 ts_tablet_manager.cc:1367] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.008s
I20240501 14:00:52.740339 7853 raft_consensus.cc:348] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.740585 7853 raft_consensus.cc:374] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.740653 7853 raft_consensus.cc:729] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.740912 7853 consensus_queue.cc:260] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.741144 5033 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:52.741788 4899 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:52.742516 7695 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:52.742741 7853 ts_tablet_manager.cc:1398] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:52.744890 7695 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:00:52.746860 7848 raft_consensus.cc:348] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.747076 7848 raft_consensus.cc:374] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:52.747148 7848 raft_consensus.cc:729] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:52.747385 7848 consensus_queue.cc:260] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.747916 7848 ts_tablet_manager.cc:1398] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:00:52.754886 2537 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:00:52.758684 7809 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 14:00:52.759480 2002 catalog_manager.cc:5484] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:52.763170 7872 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:52.764042 5033 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:52.768088 4899 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:52.772423 2005 catalog_manager.cc:5484] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:52.778391 2537 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:52.814637 7594 raft_consensus.cc:482] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:52.814850 7594 raft_consensus.cc:504] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.815656 7594 leader_election.cc:288] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:52.816147 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:52.816272 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:52.816442 4899 raft_consensus.cc:2454] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:52.816478 5161 raft_consensus.cc:2454] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:52.816833 4968 leader_election.cc:302] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:52.817102 7594 raft_consensus.cc:2781] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:52.817221 7594 raft_consensus.cc:482] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:52.817299 7594 raft_consensus.cc:3037] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.819828 7594 raft_consensus.cc:504] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.820573 7594 leader_election.cc:288] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:52.820788 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:52.820879 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:52.821018 5161 raft_consensus.cc:3037] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.821079 4899 raft_consensus.cc:3037] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:52.823405 4899 raft_consensus.cc:2454] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:52.823832 5161 raft_consensus.cc:2454] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:52.824200 4968 leader_election.cc:302] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:52.825218 7594 raft_consensus.cc:2781] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:52.825371 7594 raft_consensus.cc:686] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:52.825676 7594 consensus_queue.cc:237] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:52.831382 2002 catalog_manager.cc:5484] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:52.877794 5164 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1"
dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42238
I20240501 14:00:52.878103 5164 raft_consensus.cc:595] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Received request to transfer leadership
I20240501 14:00:52.912953 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:52.922153 5164 raft_consensus.cc:1261] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:52.922224 2537 raft_consensus.cc:1261] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:52.924624 7648 consensus_queue.cc:1035] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.925133 7648 consensus_queue.cc:1035] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:52.932484 7900 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.932972 2005 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet ee394579959049c4aaadccec3bcd0503 (table rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0]). Expected version 1 got 0
I20240501 14:00:52.933080 7900 tablet.cc:1620] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:52.933717 2002 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet ee394579959049c4aaadccec3bcd0503 (table rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0]). Expected version 1 got 0
I20240501 14:00:52.942772 7903 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.943579 7903 tablet.cc:1620] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:52.948581 5033 raft_consensus.cc:1203] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:52.949733 2002 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet ee394579959049c4aaadccec3bcd0503 (table rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0]). Expected version 1 got 0
I20240501 14:00:52.954077 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0] alter complete (version 1)
I20240501 14:00:52.954186 7901 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:52.955529 7901 tablet.cc:1620] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:52.957652 2002 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet ee394579959049c4aaadccec3bcd0503 (table rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0]). Expected version 1 got 0
I20240501 14:00:53.011015 7727 raft_consensus.cc:482] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.011207 7727 raft_consensus.cc:504] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:53.011992 7727 leader_election.cc:288] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:53.012503 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a489f3c24c54297a0569dd560cf159e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:53.012578 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a489f3c24c54297a0569dd560cf159e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:53.012751 5033 raft_consensus.cc:2454] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:53.012799 5164 raft_consensus.cc:2454] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:53.013171 2082 leader_election.cc:302] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:53.013453 7727 raft_consensus.cc:2781] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.013577 7727 raft_consensus.cc:482] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.013667 7727 raft_consensus.cc:3037] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.015712 7727 raft_consensus.cc:504] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:53.016232 7727 leader_election.cc:288] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:53.016696 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a489f3c24c54297a0569dd560cf159e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:53.016734 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a489f3c24c54297a0569dd560cf159e" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:53.016908 5164 raft_consensus.cc:3037] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.016908 5033 raft_consensus.cc:3037] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.019666 5033 raft_consensus.cc:2454] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:53.019666 5164 raft_consensus.cc:2454] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:53.020128 2080 leader_election.cc:302] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:53.020443 7727 raft_consensus.cc:2781] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.020591 7727 raft_consensus.cc:686] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:53.020855 7727 consensus_queue.cc:237] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:53.024060 2000 catalog_manager.cc:5484] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:53.062652 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:53.069859 7900 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:53.070711 7903 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:53.070986 7900 tablet.cc:1620] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:53.071177 7901 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:53.071280 7903 tablet.cc:1620] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:53.071816 7901 tablet.cc:1620] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:53.080684 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=d02883c7adb643dca6be395f9142b3c0] alter complete (version 2)
I20240501 14:00:53.110410 5164 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1"
dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42244
I20240501 14:00:53.110662 5164 raft_consensus.cc:595] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Received request to transfer leadership
I20240501 14:00:53.170780 7169 consensus_queue.cc:1035] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.181073 7865 consensus_queue.cc:1035] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.184157 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:53.192270 7900 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:53.193167 7903 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:53.197232 7901 tablet.cc:1722] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:53.202940 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=d02883c7adb643dca6be395f9142b3c0] alter complete (version 3)
I20240501 14:00:53.253644 7809 raft_consensus.cc:979] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: : Instructing follower f8c60f2bbac541dcba4424ab726f9ad9 to start an election
I20240501 14:00:53.253877 7809 raft_consensus.cc:1067] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Signalling peer f8c60f2bbac541dcba4424ab726f9ad9 to start an election
I20240501 14:00:53.254948 4899 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1"
dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
from {username='slave'} at 127.28.17.7:49731
I20240501 14:00:53.255196 4899 raft_consensus.cc:482] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:53.255321 4899 raft_consensus.cc:3037] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.257434 4899 raft_consensus.cc:504] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.258458 4899 leader_election.cc:288] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:53.258811 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:53.258906 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:53.258906 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83392c08bf314fcd98833335c247ccb1" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:53.259133 2537 raft_consensus.cc:3037] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.259143 5033 raft_consensus.cc:3037] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.260093 5164 raft_consensus.cc:3032] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:53.260185 5164 raft_consensus.cc:729] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:53.260541 5164 consensus_queue.cc:260] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.261044 5164 raft_consensus.cc:3037] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.261796 5033 raft_consensus.cc:2454] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:00:53.261796 2537 raft_consensus.cc:2454] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:00:53.262383 4836 leader_election.cc:302] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:53.262777 7869 raft_consensus.cc:2781] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:53.262915 7869 raft_consensus.cc:686] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:53.263223 7869 consensus_queue.cc:237] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.263967 5164 raft_consensus.cc:2454] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:00:53.267805 2002 catalog_manager.cc:5484] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:53.307251 2002 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:53.307502 2002 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:53.310657 2002 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet ee394579959049c4aaadccec3bcd0503
I20240501 14:00:53.311533 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet ee394579959049c4aaadccec3bcd0503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:53 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:53.311528 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet ee394579959049c4aaadccec3bcd0503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:53 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:53.311558 5012 tablet_service.cc:1514] Processing DeleteTablet for tablet ee394579959049c4aaadccec3bcd0503 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:53 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:53.312242 7927 tablet_replica.cc:330] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:53.312517 7927 raft_consensus.cc:2229] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.312803 7927 raft_consensus.cc:2258] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.313629 7928 tablet_replica.cc:330] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:53.313796 7926 tablet_replica.cc:330] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:53.313796 7927 ts_tablet_manager.cc:1875] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:53.313925 7928 raft_consensus.cc:2229] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:53.314018 7926 raft_consensus.cc:2229] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.314225 7926 raft_consensus.cc:2258] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.314716 7928 raft_consensus.cc:2258] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.315068 7926 ts_tablet_manager.cc:1875] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:53.315651 7928 ts_tablet_manager.cc:1875] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:53.321738 7926 ts_tablet_manager.cc:1888] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:53.321954 7926 log.cc:1196] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee394579959049c4aaadccec3bcd0503
I20240501 14:00:53.321990 7927 ts_tablet_manager.cc:1888] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:53.322266 7927 log.cc:1196] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/ee394579959049c4aaadccec3bcd0503
I20240501 14:00:53.322393 7926 ts_tablet_manager.cc:1909] T ee394579959049c4aaadccec3bcd0503 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:53.322813 7927 ts_tablet_manager.cc:1909] T ee394579959049c4aaadccec3bcd0503 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:53.323560 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet ee394579959049c4aaadccec3bcd0503 (table rebalancer_dynamic_table_new_name [id=d02883c7adb643dca6be395f9142b3c0]) successfully deleted
I20240501 14:00:53.323422 7928 ts_tablet_manager.cc:1888] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:53.323931 7928 log.cc:1196] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/ee394579959049c4aaadccec3bcd0503
I20240501 14:00:53.324020 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet ee394579959049c4aaadccec3bcd0503 (table rebalancer_dynamic_table_new_name [id=d02883c7adb643dca6be395f9142b3c0]) successfully deleted
I20240501 14:00:53.324414 7928 ts_tablet_manager.cc:1909] T ee394579959049c4aaadccec3bcd0503 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:53.325481 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet ee394579959049c4aaadccec3bcd0503 (table rebalancer_dynamic_table_new_name [id=d02883c7adb643dca6be395f9142b3c0]) successfully deleted
I20240501 14:00:53.337635 7902 raft_consensus.cc:1050] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:53.338380 7902 consensus_queue.cc:237] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.341284 5033 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:53.341682 2277 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:53.341691 2406 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:53.342202 7796 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:53.342790 7796 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:53.343583 7894 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:53.352039 7648 consensus_queue.cc:1035] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.368777 7648 consensus_queue.cc:1035] T 8e605c78442442dd82c598bac2fd84cd P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.404829 7894 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.408634 2274 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.420650 2000 catalog_manager.cc:5484] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.423183 2408 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.426095 5032 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.436398 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:53.446652 5012 tablet_service.cc:1467] Processing CreateTablet for tablet aa2a5c93af45437dbedd286e445b9489 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0]), partition=
I20240501 14:00:53.447080 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa2a5c93af45437dbedd286e445b9489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.447263 2258 tablet_service.cc:1467] Processing CreateTablet for tablet aa2a5c93af45437dbedd286e445b9489 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0]), partition=
I20240501 14:00:53.447573 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa2a5c93af45437dbedd286e445b9489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.451215 5276 tablet_service.cc:1467] Processing CreateTablet for tablet aa2a5c93af45437dbedd286e445b9489 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0]), partition=
I20240501 14:00:53.451553 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa2a5c93af45437dbedd286e445b9489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.459882 7727 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.464359 7949 tablet_bootstrap.cc:492] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:53.466060 7949 tablet_bootstrap.cc:654] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.471956 7881 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.479430 7947 tablet_bootstrap.cc:492] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:53.484812 7947 tablet_bootstrap.cc:654] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.499526 7947 tablet_bootstrap.cc:492] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:53.499773 7947 ts_tablet_manager.cc:1367] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240501 14:00:53.500025 7949 tablet_bootstrap.cc:492] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:53.500180 7949 ts_tablet_manager.cc:1367] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s
I20240501 14:00:53.500638 7947 raft_consensus.cc:348] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.500833 7947 raft_consensus.cc:374] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.500905 7947 raft_consensus.cc:729] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.501103 7947 consensus_queue.cc:260] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.501495 7953 tablet_bootstrap.cc:492] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:53.502142 7947 ts_tablet_manager.cc:1398] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:53.503809 7953 tablet_bootstrap.cc:654] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.506821 7949 raft_consensus.cc:348] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.507069 7949 raft_consensus.cc:374] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.507155 7949 raft_consensus.cc:729] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.508549 7949 consensus_queue.cc:260] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.510246 7953 tablet_bootstrap.cc:492] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:53.510445 7953 ts_tablet_manager.cc:1367] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240501 14:00:53.512111 7953 raft_consensus.cc:348] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.512395 7953 raft_consensus.cc:374] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.512518 7953 raft_consensus.cc:729] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.512852 7953 consensus_queue.cc:260] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.513586 7953 ts_tablet_manager.cc:1398] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:53.519919 7949 ts_tablet_manager.cc:1398] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240501 14:00:53.555307 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:53.563890 5012 tablet_service.cc:1467] Processing CreateTablet for tablet ce213e03db7f4f17913aad989c584607 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.564396 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce213e03db7f4f17913aad989c584607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.564430 5276 tablet_service.cc:1467] Processing CreateTablet for tablet ce213e03db7f4f17913aad989c584607 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.564839 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce213e03db7f4f17913aad989c584607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.565059 2518 tablet_service.cc:1467] Processing CreateTablet for tablet ce213e03db7f4f17913aad989c584607 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.565455 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce213e03db7f4f17913aad989c584607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.569103 7953 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:53.569603 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 205af5386f904e70989e647da22750e8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.569782 7594 raft_consensus.cc:482] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.570057 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205af5386f904e70989e647da22750e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.569941 7594 raft_consensus.cc:504] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.570569 7949 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:53.570817 7594 leader_election.cc:288] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:53.571553 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2a5c93af45437dbedd286e445b9489" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:53.571805 5293 raft_consensus.cc:2454] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:53.572193 7949 tablet_bootstrap.cc:654] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.572312 4969 leader_election.cc:302] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 6f81f803ec7746ccb502c59b7cbf8eb2; no voters:
I20240501 14:00:53.572646 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2a5c93af45437dbedd286e445b9489" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:53.572839 2277 raft_consensus.cc:2454] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:53.573128 5275 tablet_service.cc:1467] Processing CreateTablet for tablet cdb14f12e45440b6ba991534c9628eba (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.573280 7594 raft_consensus.cc:2781] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.573382 7594 raft_consensus.cc:482] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.573472 7594 raft_consensus.cc:3037] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.573529 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb14f12e45440b6ba991534c9628eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.574016 7953 tablet_bootstrap.cc:654] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.576839 7594 raft_consensus.cc:504] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.578239 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2a5c93af45437dbedd286e445b9489" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:53.578466 2277 raft_consensus.cc:3037] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.579617 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 205af5386f904e70989e647da22750e8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.579707 2516 tablet_service.cc:1467] Processing CreateTablet for tablet 205af5386f904e70989e647da22750e8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.580057 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205af5386f904e70989e647da22750e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.580086 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205af5386f904e70989e647da22750e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.582185 2277 raft_consensus.cc:2454] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:53.582708 4967 leader_election.cc:302] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:53.582890 2258 tablet_service.cc:1467] Processing CreateTablet for tablet cdb14f12e45440b6ba991534c9628eba (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.582911 5144 tablet_service.cc:1467] Processing CreateTablet for tablet cdb14f12e45440b6ba991534c9628eba (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:53.583284 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb14f12e45440b6ba991534c9628eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.583314 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb14f12e45440b6ba991534c9628eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:53.583823 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2a5c93af45437dbedd286e445b9489" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:53.584035 5293 raft_consensus.cc:3037] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.586654 7648 raft_consensus.cc:2781] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.586822 7648 raft_consensus.cc:686] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:53.587103 7648 consensus_queue.cc:237] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.589424 5293 raft_consensus.cc:2454] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:53.589867 7594 leader_election.cc:288] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:53.596271 7958 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:53.598282 7958 tablet_bootstrap.cc:654] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.599792 7949 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:53.600092 7949 ts_tablet_manager.cc:1367] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.006s
I20240501 14:00:53.601950 7949 raft_consensus.cc:348] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.602257 7949 raft_consensus.cc:374] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.602396 7949 raft_consensus.cc:729] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.602715 7949 consensus_queue.cc:260] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.603345 7949 ts_tablet_manager.cc:1398] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:53.604276 7961 tablet_bootstrap.cc:492] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:53.605141 7947 tablet_bootstrap.cc:492] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:53.606542 7947 tablet_bootstrap.cc:654] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.608493 7961 tablet_bootstrap.cc:654] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.611125 7958 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:53.611361 7958 ts_tablet_manager.cc:1367] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20240501 14:00:53.612361 7958 raft_consensus.cc:348] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.612681 7958 raft_consensus.cc:374] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.612790 7958 raft_consensus.cc:729] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.613384 7953 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:53.613557 7953 ts_tablet_manager.cc:1367] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.005s
I20240501 14:00:53.614775 7958 consensus_queue.cc:260] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.617115 7953 raft_consensus.cc:348] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.617679 7953 raft_consensus.cc:374] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.617915 7953 raft_consensus.cc:729] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.618108 7959 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:53.618299 7953 consensus_queue.cc:260] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.618773 2002 catalog_manager.cc:5484] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:53.621912 7953 ts_tablet_manager.cc:1398] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240501 14:00:53.623045 7953 tablet_bootstrap.cc:492] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:53.623757 7947 tablet_bootstrap.cc:492] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:53.623950 7947 ts_tablet_manager.cc:1367] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.010s
I20240501 14:00:53.624689 7947 raft_consensus.cc:348] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.625042 7947 raft_consensus.cc:374] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.625144 7958 ts_tablet_manager.cc:1398] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 14:00:53.625161 7947 raft_consensus.cc:729] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.625388 7947 consensus_queue.cc:260] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.626176 7959 tablet_bootstrap.cc:654] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.627374 7947 ts_tablet_manager.cc:1398] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:53.627030 7958 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:53.629527 7958 tablet_bootstrap.cc:654] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.629535 7953 tablet_bootstrap.cc:654] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.633641 7958 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:53.633852 7958 ts_tablet_manager.cc:1367] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240501 14:00:53.634629 7958 raft_consensus.cc:348] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.634845 7958 raft_consensus.cc:374] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.634941 7958 raft_consensus.cc:729] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.635205 7958 consensus_queue.cc:260] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.637343 7958 ts_tablet_manager.cc:1398] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:53.640377 7953 tablet_bootstrap.cc:492] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:53.640590 7953 ts_tablet_manager.cc:1367] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.009s
I20240501 14:00:53.640826 7961 tablet_bootstrap.cc:492] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:53.641016 7961 ts_tablet_manager.cc:1367] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.000s
I20240501 14:00:53.641425 7953 raft_consensus.cc:348] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.641690 7953 raft_consensus.cc:374] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.641821 7953 raft_consensus.cc:729] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.641779 7961 raft_consensus.cc:348] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.642010 7961 raft_consensus.cc:374] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.642163 7961 raft_consensus.cc:729] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.642127 7953 consensus_queue.cc:260] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.642467 7961 consensus_queue.cc:260] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.643898 7953 ts_tablet_manager.cc:1398] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:53.645717 7961 ts_tablet_manager.cc:1398] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:00:53.647410 7961 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:53.652438 7961 tablet_bootstrap.cc:654] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:53.654265 7961 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:53.654438 7961 ts_tablet_manager.cc:1367] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:00:53.656023 7961 raft_consensus.cc:348] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.656378 7961 raft_consensus.cc:374] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.656499 7961 raft_consensus.cc:729] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.656855 7961 consensus_queue.cc:260] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.658931 7961 ts_tablet_manager.cc:1398] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:00:53.665630 7894 raft_consensus.cc:482] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.665833 7894 raft_consensus.cc:504] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.666759 7894 leader_election.cc:288] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:53.667400 7959 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:53.667770 7959 ts_tablet_manager.cc:1367] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.005s
I20240501 14:00:53.667721 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205af5386f904e70989e647da22750e8" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:53.668543 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205af5386f904e70989e647da22750e8" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:53.668596 7959 raft_consensus.cc:348] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.668778 7959 raft_consensus.cc:374] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:53.668840 7959 raft_consensus.cc:729] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:53.668826 5164 raft_consensus.cc:2454] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:53.669045 7959 consensus_queue.cc:260] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.669561 2408 raft_consensus.cc:2454] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:53.669886 7959 ts_tablet_manager.cc:1398] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:53.671032 2471 leader_election.cc:302] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:53.676910 7894 raft_consensus.cc:2781] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.677063 7894 raft_consensus.cc:482] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.677145 7894 raft_consensus.cc:3037] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.683144 7894 raft_consensus.cc:504] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.684667 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205af5386f904e70989e647da22750e8" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:53.684916 2408 raft_consensus.cc:3037] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.690873 2408 raft_consensus.cc:2454] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:53.691715 2471 leader_election.cc:302] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:00:53.692018 7388 raft_consensus.cc:2781] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.692183 7388 raft_consensus.cc:686] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:53.692432 7388 consensus_queue.cc:237] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:53.695106 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205af5386f904e70989e647da22750e8" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:53.695394 5164 raft_consensus.cc:3037] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.698163 5164 raft_consensus.cc:2454] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:53.700157 7796 raft_consensus.cc:482] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.700337 7796 raft_consensus.cc:504] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.701220 7796 leader_election.cc:288] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:53.701850 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:53.701860 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:53.702065 5293 raft_consensus.cc:2454] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:53.702082 5033 raft_consensus.cc:2454] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:53.702631 2473 leader_election.cc:302] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:53.703514 7894 leader_election.cc:288] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:53.703893 7796 raft_consensus.cc:2781] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.703984 7796 raft_consensus.cc:482] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.704083 7796 raft_consensus.cc:3037] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.707438 7796 raft_consensus.cc:504] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.708199 7796 leader_election.cc:288] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:53.708635 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:53.708635 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:53.708832 5293 raft_consensus.cc:3037] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.708855 5033 raft_consensus.cc:3037] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.709416 2002 catalog_manager.cc:5484] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:53.711934 5293 raft_consensus.cc:2454] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:53.712284 5033 raft_consensus.cc:2454] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:53.712899 2473 leader_election.cc:302] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:53.713948 7388 raft_consensus.cc:2781] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.714104 7388 raft_consensus.cc:686] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:53.714385 7388 consensus_queue.cc:237] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:53.717790 2002 catalog_manager.cc:5484] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:53.731133 2538 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:00:53.731860 7618 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:53.743197 2538 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748"
dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35610
I20240501 14:00:53.743441 2538 raft_consensus.cc:595] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Received request to transfer leadership
I20240501 14:00:53.744050 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:53.749787 5293 raft_consensus.cc:1261] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:53.750335 2277 raft_consensus.cc:1261] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:53.750900 7648 consensus_queue.cc:1035] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.751174 7594 consensus_queue.cc:1035] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:53.759138 2002 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet aa2a5c93af45437dbedd286e445b9489 (table rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0]). Expected version 1 got 0
I20240501 14:00:53.761808 7991 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:53.762570 7991 tablet.cc:1620] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:53.764811 7990 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:53.765481 7990 tablet.cc:1620] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:53.765759 2002 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet aa2a5c93af45437dbedd286e445b9489 (table rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0]). Expected version 1 got 0
I20240501 14:00:53.770872 5033 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:00:53.771541 7618 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:53.773878 2002 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet aa2a5c93af45437dbedd286e445b9489 (table rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0]). Expected version 1 got 0
I20240501 14:00:53.774940 5164 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:00:53.775480 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0] alter complete (version 1)
I20240501 14:00:53.776507 7618 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:53.802846 7996 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:53.808296 1999 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet aa2a5c93af45437dbedd286e445b9489 (table rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0]). Expected version 1 got 0
I20240501 14:00:53.809484 7894 raft_consensus.cc:979] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: : Instructing follower b3da1f36f37e4a1aa9f280c2fd0e1fba to start an election
I20240501 14:00:53.809643 7894 raft_consensus.cc:1067] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Signalling peer b3da1f36f37e4a1aa9f280c2fd0e1fba to start an election
I20240501 14:00:53.810734 2274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748"
dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
from {username='slave'} at 127.28.17.4:47605
I20240501 14:00:53.812664 2274 raft_consensus.cc:482] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:53.812788 2274 raft_consensus.cc:3037] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.813038 7996 tablet.cc:1620] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:53.816808 2274 raft_consensus.cc:504] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.818192 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:53.818437 2538 raft_consensus.cc:3032] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:53.818560 2538 raft_consensus.cc:729] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:53.818869 2538 consensus_queue.cc:260] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.819334 2538 raft_consensus.cc:3037] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.822068 2538 raft_consensus.cc:2454] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 2.
W20240501 14:00:53.822520 4836 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.6:46037, user_credentials={real_user=slave}} blocked reactor thread for 46262.1us
I20240501 14:00:53.823405 2274 leader_election.cc:288] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:53.823710 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:53.823930 5033 raft_consensus.cc:3037] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.825949 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42a56dd51fe410cb42dd284ecbf2748" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:53.826165 2408 raft_consensus.cc:3037] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:53.826196 4899 consensus_queue.cc:237] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.828612 2408 raft_consensus.cc:2454] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 2.
I20240501 14:00:53.829499 2211 leader_election.cc:302] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:53.829808 7945 raft_consensus.cc:2781] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:53.829869 5031 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:00:53.829905 7945 raft_consensus.cc:686] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:53.830142 7945 consensus_queue.cc:237] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:53.830881 5033 raft_consensus.cc:2454] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 2.
I20240501 14:00:53.831039 7994 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:53.833732 1999 catalog_manager.cc:5484] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 1 to 2, leader changed from 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 2 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:53.834410 7869 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.835878 2537 raft_consensus.cc:1261] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:00:53.836550 5031 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.841354 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83392c08bf314fcd98833335c247ccb1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:53.842486 1999 catalog_manager.cc:5484] T 83392c08bf314fcd98833335c247ccb1 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 3 to 5, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.843262 7869 consensus_queue.cc:1035] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:53.846275 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 83392c08bf314fcd98833335c247ccb1 with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:53.846951 8000 tablet_replica.cc:330] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:53.847297 8000 raft_consensus.cc:2229] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.847584 8000 raft_consensus.cc:2258] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.848865 8000 ts_tablet_manager.cc:1875] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:53.858968 8000 ts_tablet_manager.cc:1888] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 14:00:53.859133 8000 log.cc:1196] T 83392c08bf314fcd98833335c247ccb1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/83392c08bf314fcd98833335c247ccb1
I20240501 14:00:53.860040 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 83392c08bf314fcd98833335c247ccb1 (table rebalancer_extra_table_8 [id=2c1ff41061c141f39952c18d0bb6d2aa]) successfully deleted
I20240501 14:00:53.860955 2537 raft_consensus.cc:2932] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:53.892777 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:53.905056 7990 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:53.905607 7990 tablet.cc:1620] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:53.905956 7996 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:53.906409 7996 tablet.cc:1620] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:53.914279 7991 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:53.914594 7872 raft_consensus.cc:482] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:53.914865 7991 tablet.cc:1620] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:53.914772 7872 raft_consensus.cc:504] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.915450 7872 leader_election.cc:288] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:53.918678 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb14f12e45440b6ba991534c9628eba" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:53.918900 2274 raft_consensus.cc:2454] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:53.919353 5097 leader_election.cc:302] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:53.919673 7872 raft_consensus.cc:2781] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:53.919767 7872 raft_consensus.cc:482] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:53.919864 7872 raft_consensus.cc:3037] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.923627 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=1a45fc1e56f844fda4569ccc6fa307d0] alter complete (version 2)
I20240501 14:00:53.924218 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb14f12e45440b6ba991534c9628eba" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:53.924448 5293 raft_consensus.cc:2454] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:53.926409 7872 raft_consensus.cc:504] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.927523 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb14f12e45440b6ba991534c9628eba" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:53.927709 5293 raft_consensus.cc:3037] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.931382 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb14f12e45440b6ba991534c9628eba" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:53.931588 2274 raft_consensus.cc:3037] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:53.934084 2274 raft_consensus.cc:2454] T cdb14f12e45440b6ba991534c9628eba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:53.934540 5097 leader_election.cc:302] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:53.934929 7695 raft_consensus.cc:2781] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:53.935047 5293 raft_consensus.cc:2454] T cdb14f12e45440b6ba991534c9628eba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:53.935107 7695 raft_consensus.cc:686] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:53.935369 7695 consensus_queue.cc:237] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:53.935607 7872 leader_election.cc:288] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:53.939273 1999 catalog_manager.cc:5484] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:53.995759 2408 consensus_queue.cc:237] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
W20240501 14:00:53.998544 2343 consensus_peers.cc:485] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01096bd6f2bf4663b70043f61ecdea5b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:53.998592 2537 raft_consensus.cc:1261] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:53.998963 2274 raft_consensus.cc:1261] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:00:53.999115 7865 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:53.999497 7169 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:00:54.003703 7279 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:54.004323 2537 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:54.004369 2274 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:00:54.009703 1999 catalog_manager.cc:5484] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 4, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:54.036258 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:54.043206 7990 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.044011 7996 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.048202 7991 tablet.cc:1722] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.055647 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=1a45fc1e56f844fda4569ccc6fa307d0] alter complete (version 3)
I20240501 14:00:54.135367 7796 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.141356 7388 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.156992 2002 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:54.157260 2002 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:54.160198 2002 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet aa2a5c93af45437dbedd286e445b9489
I20240501 14:00:54.160984 5012 tablet_service.cc:1514] Processing DeleteTablet for tablet aa2a5c93af45437dbedd286e445b9489 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:54.161003 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet aa2a5c93af45437dbedd286e445b9489 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:54.161465 8036 tablet_replica.cc:330] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:54.161921 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet aa2a5c93af45437dbedd286e445b9489 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:54.162719 8036 raft_consensus.cc:2229] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:54.162792 8038 tablet_replica.cc:330] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:54.162993 8036 raft_consensus.cc:2258] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.163017 8038 raft_consensus.cc:2229] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:54.163239 8038 raft_consensus.cc:2258] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.164201 8038 ts_tablet_manager.cc:1875] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.166767 8036 ts_tablet_manager.cc:1875] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.166805 8037 tablet_replica.cc:330] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:54.166997 8037 raft_consensus.cc:2229] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:54.167317 8037 raft_consensus.cc:2258] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.168169 8037 ts_tablet_manager.cc:1875] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.171111 8038 ts_tablet_manager.cc:1888] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:54.171308 8038 log.cc:1196] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/aa2a5c93af45437dbedd286e445b9489
I20240501 14:00:54.171988 8038 ts_tablet_manager.cc:1909] T aa2a5c93af45437dbedd286e445b9489 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:54.172618 8037 ts_tablet_manager.cc:1888] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:54.172739 8037 log.cc:1196] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/aa2a5c93af45437dbedd286e445b9489
I20240501 14:00:54.173056 8036 ts_tablet_manager.cc:1888] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:54.173198 8036 log.cc:1196] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa2a5c93af45437dbedd286e445b9489
I20240501 14:00:54.173215 8037 ts_tablet_manager.cc:1909] T aa2a5c93af45437dbedd286e445b9489 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:54.173224 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet aa2a5c93af45437dbedd286e445b9489 (table rebalancer_dynamic_table_new_name [id=1a45fc1e56f844fda4569ccc6fa307d0]) successfully deleted
I20240501 14:00:54.173528 8036 ts_tablet_manager.cc:1909] T aa2a5c93af45437dbedd286e445b9489 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:54.174180 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet aa2a5c93af45437dbedd286e445b9489 (table rebalancer_dynamic_table_new_name [id=1a45fc1e56f844fda4569ccc6fa307d0]) successfully deleted
I20240501 14:00:54.174517 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet aa2a5c93af45437dbedd286e445b9489 (table rebalancer_dynamic_table_new_name [id=1a45fc1e56f844fda4569ccc6fa307d0]) successfully deleted
I20240501 14:00:54.240702 2537 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:00:54.241267 7945 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:54.263864 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:54.266963 7796 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.267356 5033 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:00:54.268204 7945 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:54.277046 7894 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.277552 2406 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:00:54.278851 7862 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:54.283852 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 91a16dd581f946ac9feb62a39040bede (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]), partition=
I20240501 14:00:54.284189 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a16dd581f946ac9feb62a39040bede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.286902 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 91a16dd581f946ac9feb62a39040bede (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]), partition=
I20240501 14:00:54.287340 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a16dd581f946ac9feb62a39040bede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.292276 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 91a16dd581f946ac9feb62a39040bede (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]), partition=
I20240501 14:00:54.292646 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a16dd581f946ac9feb62a39040bede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.294535 8052 tablet_bootstrap.cc:492] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:54.296350 8052 tablet_bootstrap.cc:654] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.298604 8052 tablet_bootstrap.cc:492] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:54.298822 8052 ts_tablet_manager.cc:1367] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 14:00:54.300817 8052 raft_consensus.cc:348] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.301100 8052 raft_consensus.cc:374] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.301275 8052 raft_consensus.cc:729] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.301615 8052 consensus_queue.cc:260] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.302421 8052 ts_tablet_manager.cc:1398] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:54.312177 8054 tablet_bootstrap.cc:492] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:54.313963 8054 tablet_bootstrap.cc:654] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.316023 8054 tablet_bootstrap.cc:492] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:54.316181 8054 ts_tablet_manager.cc:1367] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:54.316793 8054 raft_consensus.cc:348] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.316988 8054 raft_consensus.cc:374] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.317080 8054 raft_consensus.cc:729] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.317294 8054 consensus_queue.cc:260] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.317757 8054 ts_tablet_manager.cc:1398] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:54.318436 7905 raft_consensus.cc:482] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.318625 7905 raft_consensus.cc:504] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.319401 7905 leader_election.cc:288] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:54.319716 8055 tablet_bootstrap.cc:492] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:54.320159 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a16dd581f946ac9feb62a39040bede" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:00:54.320881 4968 leader_election.cc:341] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:54.321138 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a16dd581f946ac9feb62a39040bede" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:54.321388 5164 raft_consensus.cc:2454] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:54.321565 8055 tablet_bootstrap.cc:654] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.321759 4967 leader_election.cc:302] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:54.322186 7905 raft_consensus.cc:2781] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:54.322345 7905 raft_consensus.cc:482] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.322454 7905 raft_consensus.cc:3037] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.325845 8055 tablet_bootstrap.cc:492] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:54.326011 8055 ts_tablet_manager.cc:1367] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:00:54.325911 7905 raft_consensus.cc:504] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.326871 8055 raft_consensus.cc:348] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.327253 8055 raft_consensus.cc:374] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.327374 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a16dd581f946ac9feb62a39040bede" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:54.327399 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a16dd581f946ac9feb62a39040bede" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:54.327421 8055 raft_consensus.cc:729] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.327593 5164 raft_consensus.cc:3037] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.327759 8055 consensus_queue.cc:260] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.328332 8055 ts_tablet_manager.cc:1398] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:54.328574 4899 raft_consensus.cc:3037] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.329936 5164 raft_consensus.cc:2454] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:54.330287 4967 leader_election.cc:302] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:54.330518 7905 leader_election.cc:288] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:54.330612 7594 raft_consensus.cc:2781] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.330724 7594 raft_consensus.cc:686] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:54.330971 7594 consensus_queue.cc:237] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:54.332326 4899 raft_consensus.cc:2454] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:54.336411 2005 catalog_manager.cc:5484] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:54.381060 8059 ts_tablet_manager.cc:902] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:54.383266 8059 tablet_copy_client.cc:308] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:00:54.384030 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet 01096bd6f2bf4663b70043f61ecdea5b from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:41439)
I20240501 14:00:54.384212 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet 01096bd6f2bf4663b70043f61ecdea5b from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:41439: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-01096bd6f2bf4663b70043f61ecdea5b
I20240501 14:00:54.386441 2428 tablet_copy_source_session.cc:205] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:54.387640 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01096bd6f2bf4663b70043f61ecdea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.392778 8059 tablet_copy_client.cc:783] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:54.393049 8059 tablet_copy_client.cc:647] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:54.394970 8059 tablet_copy_client.cc:523] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:54.397922 8059 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:54.430138 7695 consensus_queue.cc:1035] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.433018 8059 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:54.435210 7872 consensus_queue.cc:1035] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.450675 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:54.451740 8059 tablet_bootstrap.cc:492] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:00:54.452064 8059 ts_tablet_manager.cc:1367] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.054s user 0.033s sys 0.005s
I20240501 14:00:54.452781 8059 raft_consensus.cc:348] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:54.453080 8059 raft_consensus.cc:729] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:00:54.453302 8059 consensus_queue.cc:260] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:00:54.453859 8059 ts_tablet_manager.cc:1398] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:54.454639 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-01096bd6f2bf4663b70043f61ecdea5b received from {username='slave'} at 127.28.17.6:41439
I20240501 14:00:54.454771 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-01096bd6f2bf4663b70043f61ecdea5b on tablet 01096bd6f2bf4663b70043f61ecdea5b with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:54.457520 4899 raft_consensus.cc:1261] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:54.457520 5164 raft_consensus.cc:1261] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:54.458077 7648 consensus_queue.cc:1035] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:54.459106 7648 consensus_queue.cc:1035] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 14:00:54.466583 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:54.470352 7990 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.470960 7990 tablet.cc:1620] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:54.473017 2005 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]). Expected version 1 got 0
I20240501 14:00:54.473609 2002 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]). Expected version 1 got 0
I20240501 14:00:54.480789 8073 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.482193 8073 tablet.cc:1620] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:54.483938 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 8e52258f0b8f4bcdaebc75618bd2baa3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.484423 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e52258f0b8f4bcdaebc75618bd2baa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.484933 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 8e52258f0b8f4bcdaebc75618bd2baa3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.485309 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e52258f0b8f4bcdaebc75618bd2baa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.486889 5275 tablet_service.cc:1467] Processing CreateTablet for tablet ccec6fa18f844ace900db8bfe91738a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.487314 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccec6fa18f844ace900db8bfe91738a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.487829 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 8e52258f0b8f4bcdaebc75618bd2baa3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.488199 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e52258f0b8f4bcdaebc75618bd2baa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.488993 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 46abd9f84a1c45f4b54447990d8ba65b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.489342 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46abd9f84a1c45f4b54447990d8ba65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.491910 2387 tablet_service.cc:1467] Processing CreateTablet for tablet ccec6fa18f844ace900db8bfe91738a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.492190 5142 tablet_service.cc:1467] Processing CreateTablet for tablet ccec6fa18f844ace900db8bfe91738a5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.492338 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccec6fa18f844ace900db8bfe91738a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.492561 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccec6fa18f844ace900db8bfe91738a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.494872 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 46abd9f84a1c45f4b54447990d8ba65b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.495272 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46abd9f84a1c45f4b54447990d8ba65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.499800 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 46abd9f84a1c45f4b54447990d8ba65b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:54.500278 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46abd9f84a1c45f4b54447990d8ba65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.502113 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd] alter complete (version 1)
I20240501 14:00:54.503845 1999 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]). Expected version 1 got 0
I20240501 14:00:54.506632 8074 tablet_bootstrap.cc:492] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:54.511518 8074 tablet_bootstrap.cc:654] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.515594 8074 tablet_bootstrap.cc:492] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:54.519034 8074 ts_tablet_manager.cc:1367] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 14:00:54.520210 8074 raft_consensus.cc:348] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.520541 8074 raft_consensus.cc:374] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.520653 8074 raft_consensus.cc:729] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.520902 8074 consensus_queue.cc:260] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.521559 8074 ts_tablet_manager.cc:1398] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:54.526463 8074 tablet_bootstrap.cc:492] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:54.528013 8074 tablet_bootstrap.cc:654] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.533905 8074 tablet_bootstrap.cc:492] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:54.534186 8074 ts_tablet_manager.cc:1367] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240501 14:00:54.534942 8074 raft_consensus.cc:348] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.535399 8074 raft_consensus.cc:374] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.535598 8074 raft_consensus.cc:729] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.536187 8055 tablet_bootstrap.cc:492] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:54.538435 8055 tablet_bootstrap.cc:654] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.539393 8074 consensus_queue.cc:260] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.540001 8074 ts_tablet_manager.cc:1398] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:00:54.548933 8054 tablet_bootstrap.cc:492] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:54.550726 8054 tablet_bootstrap.cc:654] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.550747 8083 tablet_bootstrap.cc:492] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:54.561285 8052 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:54.563063 8052 tablet_bootstrap.cc:654] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.558096 8082 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:54.567334 8082 tablet_bootstrap.cc:654] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.569914 8078 tablet_bootstrap.cc:492] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:54.573669 2002 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]). Expected version 1 got 0
I20240501 14:00:54.576723 8078 tablet_bootstrap.cc:654] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.579555 8078 tablet_bootstrap.cc:492] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:54.579802 8078 ts_tablet_manager.cc:1367] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:00:54.581410 8078 raft_consensus.cc:348] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.582227 8078 raft_consensus.cc:374] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.582383 8078 raft_consensus.cc:729] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.582404 8089 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:54.583143 8089 tablet.cc:1620] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:54.584462 8078 consensus_queue.cc:260] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.586215 8078 ts_tablet_manager.cc:1398] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:54.587978 8082 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:54.587965 2005 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]). Expected version 1 got 0
I20240501 14:00:54.588172 8052 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:54.588297 8082 ts_tablet_manager.cc:1367] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20240501 14:00:54.588353 8052 ts_tablet_manager.cc:1367] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.001s
I20240501 14:00:54.589170 8082 raft_consensus.cc:348] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.589140 8052 raft_consensus.cc:348] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.589390 8052 raft_consensus.cc:374] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.589460 8052 raft_consensus.cc:729] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.589581 8082 raft_consensus.cc:374] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.589677 8082 raft_consensus.cc:729] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.589694 8052 consensus_queue.cc:260] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.590265 8052 ts_tablet_manager.cc:1398] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:54.591303 8032 raft_consensus.cc:482] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.591435 8032 raft_consensus.cc:504] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.592061 8032 leader_election.cc:288] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:54.592316 7862 raft_consensus.cc:482] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.592428 8083 tablet_bootstrap.cc:654] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.592442 7862 raft_consensus.cc:504] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.592804 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46abd9f84a1c45f4b54447990d8ba65b" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:54.592825 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46abd9f84a1c45f4b54447990d8ba65b" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:54.593230 7862 leader_election.cc:288] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:00:54.593367 2081 leader_election.cc:341] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:54.594242 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e52258f0b8f4bcdaebc75618bd2baa3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:54.594439 2146 raft_consensus.cc:2454] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:54.594885 2211 leader_election.cc:302] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:54.595175 7862 raft_consensus.cc:2781] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 14:00:54.595217 2080 leader_election.cc:341] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:54.595260 7862 raft_consensus.cc:482] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.595319 7862 raft_consensus.cc:3037] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.595324 2080 leader_election.cc:302] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:00:54.595520 8082 consensus_queue.cc:260] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.596284 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e52258f0b8f4bcdaebc75618bd2baa3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:54.596432 8082 ts_tablet_manager.cc:1398] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:00:54.596788 8032 raft_consensus.cc:2726] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 14:00:54.597154 2213 leader_election.cc:341] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:54.597916 7862 raft_consensus.cc:504] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.599738 7862 leader_election.cc:288] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:54.599902 8055 tablet_bootstrap.cc:492] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:54.600070 8055 ts_tablet_manager.cc:1367] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.008s
I20240501 14:00:54.600033 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e52258f0b8f4bcdaebc75618bd2baa3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:54.600224 2146 raft_consensus.cc:3037] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.600169 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e52258f0b8f4bcdaebc75618bd2baa3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
W20240501 14:00:54.600708 2213 leader_election.cc:341] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:54.601059 8055 raft_consensus.cc:348] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.601361 8055 raft_consensus.cc:374] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.601485 8055 raft_consensus.cc:729] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.601786 8055 consensus_queue.cc:260] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.602911 2146 raft_consensus.cc:2454] T 8e52258f0b8f4bcdaebc75618bd2baa3 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:54.603380 2211 leader_election.cc:302] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:00:54.603816 7862 raft_consensus.cc:2781] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.604240 7862 raft_consensus.cc:686] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:54.604277 8055 ts_tablet_manager.cc:1398] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:00:54.604502 7862 consensus_queue.cc:237] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.606343 8054 tablet_bootstrap.cc:492] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:54.610335 8054 ts_tablet_manager.cc:1367] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.010s
I20240501 14:00:54.610920 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:54.613312 8083 tablet_bootstrap.cc:492] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:54.613481 8083 ts_tablet_manager.cc:1367] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20240501 14:00:54.614230 8083 raft_consensus.cc:348] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.614434 8083 raft_consensus.cc:374] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.614676 8083 raft_consensus.cc:729] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.614969 8083 consensus_queue.cc:260] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.615286 8054 raft_consensus.cc:348] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.615552 8083 ts_tablet_manager.cc:1398] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:54.615864 8083 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:54.615887 8054 raft_consensus.cc:374] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.616103 8054 raft_consensus.cc:729] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.616596 8054 consensus_queue.cc:260] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:54.618427 8083 tablet_bootstrap.cc:654] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:54.624274 2005 catalog_manager.cc:5484] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:54.626261 7990 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.626848 7990 tablet.cc:1620] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:54.626984 8073 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.627434 8073 tablet.cc:1620] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:54.628239 8054 ts_tablet_manager.cc:1398] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:00:54.630864 7279 raft_consensus.cc:482] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.631057 7279 raft_consensus.cc:504] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.631971 7279 leader_election.cc:288] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:54.633464 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46abd9f84a1c45f4b54447990d8ba65b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:54.633679 2146 raft_consensus.cc:2454] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:54.634447 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46abd9f84a1c45f4b54447990d8ba65b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:54.634691 4901 raft_consensus.cc:2454] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 0.
I20240501 14:00:54.635169 2341 leader_election.cc:302] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:54.635836 2005 catalog_manager.cc:5576] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059) does not have the latest schema for tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd]). Expected version 2 got 1
I20240501 14:00:54.637091 8089 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:54.637599 8089 tablet.cc:1620] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:54.637609 7279 raft_consensus.cc:2781] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:54.637709 7279 raft_consensus.cc:482] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.637802 7279 raft_consensus.cc:3037] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.640547 7279 raft_consensus.cc:504] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.644317 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=9804721544c9460d84afbf5544ed03dd] alter complete (version 2)
I20240501 14:00:54.647390 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46abd9f84a1c45f4b54447990d8ba65b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:54.647617 2146 raft_consensus.cc:3037] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.649664 2146 raft_consensus.cc:2454] T 46abd9f84a1c45f4b54447990d8ba65b P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:54.650709 2341 leader_election.cc:302] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:54.651042 7865 raft_consensus.cc:2781] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.651489 7865 raft_consensus.cc:686] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:00:54.651857 7865 consensus_queue.cc:237] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:54.654358 7279 leader_election.cc:288] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:00:54.655687 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46abd9f84a1c45f4b54447990d8ba65b" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:54.655934 4899 raft_consensus.cc:3037] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.656993 2002 catalog_manager.cc:5484] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:54.658826 4899 raft_consensus.cc:2454] T 46abd9f84a1c45f4b54447990d8ba65b P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:00:54.670537 8083 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:54.672359 8083 ts_tablet_manager.cc:1367] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.008s
I20240501 14:00:54.674052 8083 raft_consensus.cc:348] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.676841 8083 raft_consensus.cc:374] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:54.676959 8083 raft_consensus.cc:729] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:54.677237 8083 consensus_queue.cc:260] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.677865 8083 ts_tablet_manager.cc:1398] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:00:54.734009 7695 raft_consensus.cc:482] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:54.734228 7695 raft_consensus.cc:504] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.734872 7695 leader_election.cc:288] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:54.735459 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccec6fa18f844ace900db8bfe91738a5" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:54.735603 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccec6fa18f844ace900db8bfe91738a5" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:54.735707 2406 raft_consensus.cc:2454] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:54.735841 5293 raft_consensus.cc:2454] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:54.736168 5097 leader_election.cc:302] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:54.736514 7695 raft_consensus.cc:2781] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:54.736614 7695 raft_consensus.cc:482] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:54.736708 7695 raft_consensus.cc:3037] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.739106 7695 raft_consensus.cc:504] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.739792 7695 leader_election.cc:288] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:54.740167 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccec6fa18f844ace900db8bfe91738a5" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:54.740257 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccec6fa18f844ace900db8bfe91738a5" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:54.740366 5293 raft_consensus.cc:3037] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.740455 2408 raft_consensus.cc:3037] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:54.742311 5293 raft_consensus.cc:2454] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:54.742467 2408 raft_consensus.cc:2454] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:54.742826 5099 leader_election.cc:302] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:54.743113 7695 raft_consensus.cc:2781] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:54.743337 7695 raft_consensus.cc:686] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:54.743582 7695 consensus_queue.cc:237] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:54.747064 2002 catalog_manager.cc:5484] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:54.752774 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:54.759382 7990 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.760452 8089 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.763236 8073 tablet.cc:1722] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:54.771003 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=9804721544c9460d84afbf5544ed03dd] alter complete (version 3)
I20240501 14:00:54.858461 2274 consensus_queue.cc:237] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:54.860251 2408 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:00:54.860327 5033 raft_consensus.cc:1261] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:00:54.860962 8100 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:00:54.861274 7862 consensus_queue.cc:1035] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:00:54.863515 8100 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:54.864248 2408 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:54.864959 5033 raft_consensus.cc:2932] T d42a56dd51fe410cb42dd284ecbf2748 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:54.866845 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d42a56dd51fe410cb42dd284ecbf2748 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:54.870939 2005 catalog_manager.cc:5484] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 5 to 7, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 2 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:54.875171 2000 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:54.875416 2000 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:54.876224 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet d42a56dd51fe410cb42dd284ecbf2748 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:54.877055 8101 tablet_replica.cc:330] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:54.877301 8101 raft_consensus.cc:2229] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:54.877499 8101 raft_consensus.cc:2258] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.878356 8101 ts_tablet_manager.cc:1875] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:54.880095 2000 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 91a16dd581f946ac9feb62a39040bede
I20240501 14:00:54.881379 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a16dd581f946ac9feb62a39040bede with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:54.881456 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a16dd581f946ac9feb62a39040bede with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:54.883898 5012 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a16dd581f946ac9feb62a39040bede with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:54 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:54.892711 8103 tablet_replica.cc:330] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:54.894770 8103 raft_consensus.cc:2229] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:54.895108 8103 raft_consensus.cc:2258] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.896149 8103 ts_tablet_manager.cc:1875] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.907227 8103 ts_tablet_manager.cc:1888] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:54.907321 8104 tablet_replica.cc:330] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:54.907562 8103 log.cc:1196] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/91a16dd581f946ac9feb62a39040bede
I20240501 14:00:54.908210 8102 tablet_replica.cc:330] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:54.908193 8104 raft_consensus.cc:2229] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:54.908552 8104 raft_consensus.cc:2258] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.908547 8102 raft_consensus.cc:2229] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:54.908779 8102 raft_consensus.cc:2258] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:54.910637 8104 ts_tablet_manager.cc:1875] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.910832 8103 ts_tablet_manager.cc:1909] T 91a16dd581f946ac9feb62a39040bede P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:54.912192 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table_new_name [id=9804721544c9460d84afbf5544ed03dd]) successfully deleted
I20240501 14:00:54.914634 8101 ts_tablet_manager.cc:1888] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:00:54.917042 8102 ts_tablet_manager.cc:1875] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:54.920858 8104 ts_tablet_manager.cc:1888] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:54.924660 8104 log.cc:1196] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/91a16dd581f946ac9feb62a39040bede
I20240501 14:00:54.926064 8102 ts_tablet_manager.cc:1888] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:54.926211 8104 ts_tablet_manager.cc:1909] T 91a16dd581f946ac9feb62a39040bede P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:54.926283 8102 log.cc:1196] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/91a16dd581f946ac9feb62a39040bede
I20240501 14:00:54.926524 8101 log.cc:1196] T d42a56dd51fe410cb42dd284ecbf2748 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/d42a56dd51fe410cb42dd284ecbf2748
I20240501 14:00:54.926927 8102 ts_tablet_manager.cc:1909] T 91a16dd581f946ac9feb62a39040bede P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:54.927541 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet d42a56dd51fe410cb42dd284ecbf2748 (table rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]) successfully deleted
I20240501 14:00:54.928030 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table_new_name [id=9804721544c9460d84afbf5544ed03dd]) successfully deleted
I20240501 14:00:54.929823 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 91a16dd581f946ac9feb62a39040bede (table rebalancer_dynamic_table_new_name [id=9804721544c9460d84afbf5544ed03dd]) successfully deleted
I20240501 14:00:54.953071 5033 raft_consensus.cc:1203] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:00:54.984601 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:54.992863 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 7dac68f3bd2a4608ba106973c8d55de1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]), partition=
I20240501 14:00:54.992910 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 7dac68f3bd2a4608ba106973c8d55de1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]), partition=
I20240501 14:00:54.993232 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dac68f3bd2a4608ba106973c8d55de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.993259 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dac68f3bd2a4608ba106973c8d55de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.995471 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 7dac68f3bd2a4608ba106973c8d55de1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]), partition=
I20240501 14:00:54.995841 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dac68f3bd2a4608ba106973c8d55de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.999300 8120 tablet_bootstrap.cc:492] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:55.001531 8120 tablet_bootstrap.cc:654] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.003206 5033 raft_consensus.cc:3037] T 8e52258f0b8f4bcdaebc75618bd2baa3 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.003597 8120 tablet_bootstrap.cc:492] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:55.003741 8120 ts_tablet_manager.cc:1367] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:00:55.004513 8120 raft_consensus.cc:348] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.004720 8120 raft_consensus.cc:374] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.004796 8120 raft_consensus.cc:729] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.005098 8120 consensus_queue.cc:260] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.006306 8100 consensus_queue.cc:1035] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.007784 8120 ts_tablet_manager.cc:1398] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:00:55.010751 8052 tablet_bootstrap.cc:492] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:55.012462 8052 tablet_bootstrap.cc:654] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.018800 7862 consensus_queue.cc:1035] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.021986 8054 tablet_bootstrap.cc:492] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:55.024056 8054 tablet_bootstrap.cc:654] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.028515 8052 tablet_bootstrap.cc:492] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:55.028514 8054 tablet_bootstrap.cc:492] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:55.028685 8054 ts_tablet_manager.cc:1367] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:00:55.028781 8052 ts_tablet_manager.cc:1367] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240501 14:00:55.029639 8052 raft_consensus.cc:348] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.029732 8054 raft_consensus.cc:348] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.029933 8054 raft_consensus.cc:374] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.029958 8052 raft_consensus.cc:374] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.030054 8054 raft_consensus.cc:729] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.030105 8052 raft_consensus.cc:729] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.030364 8054 consensus_queue.cc:260] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.030422 8052 consensus_queue.cc:260] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.030992 8054 ts_tablet_manager.cc:1398] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:55.038134 8052 ts_tablet_manager.cc:1398] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:00:55.142109 7872 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.149322 7809 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.186540 7865 consensus_queue.cc:1035] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.197840 7865 consensus_queue.cc:1035] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.285156 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:55.286908 7809 raft_consensus.cc:482] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.287091 7809 raft_consensus.cc:504] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.287868 7809 leader_election.cc:288] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:55.288224 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dac68f3bd2a4608ba106973c8d55de1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:55.288465 2537 raft_consensus.cc:2454] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:55.288817 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dac68f3bd2a4608ba106973c8d55de1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:55.289073 5033 raft_consensus.cc:2454] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:55.289106 5097 leader_election.cc:302] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:55.289750 7809 raft_consensus.cc:2781] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.289873 7809 raft_consensus.cc:482] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.289963 7809 raft_consensus.cc:3037] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.292593 7809 raft_consensus.cc:504] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.293327 7809 leader_election.cc:288] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:55.294023 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dac68f3bd2a4608ba106973c8d55de1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:55.294241 2537 raft_consensus.cc:3037] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.294757 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dac68f3bd2a4608ba106973c8d55de1" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:55.294998 5033 raft_consensus.cc:3037] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.295220 4881 tablet_service.cc:1467] Processing CreateTablet for tablet ead586b9a8024a218c55d2bb579a16de (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.295624 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead586b9a8024a218c55d2bb579a16de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.296505 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 782c96bf253c4b09bc9c693d2a553b00 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.296921 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782c96bf253c4b09bc9c693d2a553b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.297737 5033 raft_consensus.cc:2454] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:55.299180 2518 tablet_service.cc:1467] Processing CreateTablet for tablet ead586b9a8024a218c55d2bb579a16de (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.299306 2258 tablet_service.cc:1467] Processing CreateTablet for tablet ead586b9a8024a218c55d2bb579a16de (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.299623 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead586b9a8024a218c55d2bb579a16de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.299721 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead586b9a8024a218c55d2bb579a16de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.300495 2537 raft_consensus.cc:2454] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:55.301025 5097 leader_election.cc:302] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:55.301381 7809 raft_consensus.cc:2781] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.301519 7809 raft_consensus.cc:686] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:55.301786 7809 consensus_queue.cc:237] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:55.302870 2255 tablet_service.cc:1467] Processing CreateTablet for tablet bc808ec4077a42d186aba7d4af5ac158 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.303273 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc808ec4077a42d186aba7d4af5ac158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.306811 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 782c96bf253c4b09bc9c693d2a553b00 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.307062 8120 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:55.307207 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782c96bf253c4b09bc9c693d2a553b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.308656 8120 tablet_bootstrap.cc:654] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.309711 8141 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:55.311347 2388 tablet_service.cc:1467] Processing CreateTablet for tablet bc808ec4077a42d186aba7d4af5ac158 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.311762 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc808ec4077a42d186aba7d4af5ac158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.311784 8141 tablet_bootstrap.cc:654] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.314643 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 782c96bf253c4b09bc9c693d2a553b00 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.315062 8141 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:55.315064 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782c96bf253c4b09bc9c693d2a553b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.315186 8141 ts_tablet_manager.cc:1367] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:00:55.315845 8141 raft_consensus.cc:348] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.316009 8141 raft_consensus.cc:374] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.316087 8141 raft_consensus.cc:729] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.316392 8141 consensus_queue.cc:260] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.317903 8141 ts_tablet_manager.cc:1398] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:55.320858 8144 tablet_bootstrap.cc:492] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:55.322366 8144 tablet_bootstrap.cc:654] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.324407 5142 tablet_service.cc:1467] Processing CreateTablet for tablet bc808ec4077a42d186aba7d4af5ac158 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:55.324892 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc808ec4077a42d186aba7d4af5ac158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.328341 8144 tablet_bootstrap.cc:492] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:55.328526 8144 ts_tablet_manager.cc:1367] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 14:00:55.328749 2005 catalog_manager.cc:5484] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:55.329032 8144 raft_consensus.cc:348] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.329610 8144 raft_consensus.cc:374] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.329674 8144 raft_consensus.cc:729] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.331255 8144 consensus_queue.cc:260] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.331881 8144 ts_tablet_manager.cc:1398] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:55.341564 8146 tablet_bootstrap.cc:492] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:55.344085 8146 tablet_bootstrap.cc:654] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.350579 8054 tablet_bootstrap.cc:492] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:55.352226 8054 tablet_bootstrap.cc:654] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.361718 8141 tablet_bootstrap.cc:492] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:55.362185 8146 tablet_bootstrap.cc:492] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:55.362367 8146 ts_tablet_manager.cc:1367] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.010s
I20240501 14:00:55.363350 8146 raft_consensus.cc:348] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.363986 8146 raft_consensus.cc:374] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.364069 8146 raft_consensus.cc:729] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.364316 8146 consensus_queue.cc:260] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.364751 8141 tablet_bootstrap.cc:654] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.366025 8146 ts_tablet_manager.cc:1398] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:55.366322 8146 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:55.368129 8146 tablet_bootstrap.cc:654] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.371201 8141 tablet_bootstrap.cc:492] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:55.371526 8141 ts_tablet_manager.cc:1367] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240501 14:00:55.372439 8141 raft_consensus.cc:348] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.372735 8141 raft_consensus.cc:374] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.372864 8141 raft_consensus.cc:729] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.373167 8141 consensus_queue.cc:260] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.373776 8141 ts_tablet_manager.cc:1398] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:55.374570 8052 tablet_bootstrap.cc:492] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:55.376518 8054 tablet_bootstrap.cc:492] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:55.376611 8052 tablet_bootstrap.cc:654] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.376711 8054 ts_tablet_manager.cc:1367] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20240501 14:00:55.377566 8054 raft_consensus.cc:348] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.377852 8054 raft_consensus.cc:374] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.377920 8054 raft_consensus.cc:729] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.380383 8054 consensus_queue.cc:260] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.383939 8054 ts_tablet_manager.cc:1398] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 14:00:55.401450 8052 tablet_bootstrap.cc:492] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:55.401772 8052 ts_tablet_manager.cc:1367] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240501 14:00:55.409991 8052 raft_consensus.cc:348] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.411159 8052 raft_consensus.cc:374] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.411311 8052 raft_consensus.cc:729] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.411653 8052 consensus_queue.cc:260] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.413861 8052 ts_tablet_manager.cc:1398] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:00:55.415427 8146 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:55.415869 8146 ts_tablet_manager.cc:1367] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.014s
I20240501 14:00:55.416924 8146 raft_consensus.cc:348] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.418191 8146 raft_consensus.cc:374] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.418457 8146 raft_consensus.cc:729] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.419008 8146 consensus_queue.cc:260] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.421403 8146 ts_tablet_manager.cc:1398] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:55.422188 8146 tablet_bootstrap.cc:492] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:55.424751 8146 tablet_bootstrap.cc:654] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:55.435809 8146 tablet_bootstrap.cc:492] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:55.435807 8120 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:55.436093 8146 ts_tablet_manager.cc:1367] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240501 14:00:55.436864 8146 raft_consensus.cc:348] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.436051 8120 ts_tablet_manager.cc:1367] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.129s user 0.048s sys 0.044s
I20240501 14:00:55.438815 8120 raft_consensus.cc:348] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.439038 8120 raft_consensus.cc:374] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.439097 8120 raft_consensus.cc:729] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.439333 8120 consensus_queue.cc:260] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.441442 8120 ts_tablet_manager.cc:1398] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:55.443373 8146 raft_consensus.cc:374] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:55.445748 8146 raft_consensus.cc:729] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:55.446242 8146 consensus_queue.cc:260] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.446851 8146 ts_tablet_manager.cc:1398] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 14:00:55.456974 8100 raft_consensus.cc:482] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.457183 8100 raft_consensus.cc:504] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.458022 8100 leader_election.cc:288] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:55.458678 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782c96bf253c4b09bc9c693d2a553b00" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:55.458904 5033 raft_consensus.cc:2454] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:55.458833 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782c96bf253c4b09bc9c693d2a553b00" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:55.459064 4901 raft_consensus.cc:2454] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:55.459535 2212 leader_election.cc:302] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:55.461472 8100 raft_consensus.cc:2781] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.461589 8100 raft_consensus.cc:482] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.461678 8100 raft_consensus.cc:3037] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.464269 8100 raft_consensus.cc:504] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.465382 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782c96bf253c4b09bc9c693d2a553b00" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:55.465601 4901 raft_consensus.cc:3037] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.468331 4901 raft_consensus.cc:2454] T 782c96bf253c4b09bc9c693d2a553b00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:55.468693 8100 leader_election.cc:288] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:55.469122 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782c96bf253c4b09bc9c693d2a553b00" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:55.469168 2212 leader_election.cc:302] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:55.469316 5033 raft_consensus.cc:3037] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.469457 7862 raft_consensus.cc:2781] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.469584 7862 raft_consensus.cc:686] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:55.469839 7862 consensus_queue.cc:237] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:55.472023 5033 raft_consensus.cc:2454] T 782c96bf253c4b09bc9c693d2a553b00 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:55.473930 2000 catalog_manager.cc:5484] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:55.482734 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:55.484822 8154 raft_consensus.cc:482] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.485003 8154 raft_consensus.cc:504] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.485822 8154 leader_election.cc:288] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:55.486438 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead586b9a8024a218c55d2bb579a16de" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:55.486704 2274 raft_consensus.cc:2454] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:55.487239 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead586b9a8024a218c55d2bb579a16de" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:00:55.487385 2471 leader_election.cc:302] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:55.487510 4899 raft_consensus.cc:2454] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:55.487902 8154 raft_consensus.cc:2781] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.488013 8154 raft_consensus.cc:482] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.488085 8154 raft_consensus.cc:3037] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.489969 8154 raft_consensus.cc:504] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.491012 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead586b9a8024a218c55d2bb579a16de" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:00:55.491232 4899 raft_consensus.cc:3037] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.493113 4899 raft_consensus.cc:2454] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:55.493541 2472 leader_election.cc:302] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:55.493537 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead586b9a8024a218c55d2bb579a16de" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:55.493697 2274 raft_consensus.cc:3037] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.493857 7388 raft_consensus.cc:2781] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.494266 7388 raft_consensus.cc:686] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:55.494599 7388 consensus_queue.cc:237] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:55.496260 2274 raft_consensus.cc:2454] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:55.496966 8154 leader_election.cc:288] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:55.498644 2005 catalog_manager.cc:5484] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:55.501776 5033 raft_consensus.cc:1261] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:55.502597 7872 consensus_queue.cc:1035] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.502800 2537 raft_consensus.cc:1261] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:55.503346 7872 consensus_queue.cc:1035] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.509673 2005 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]). Expected version 1 got 0
I20240501 14:00:55.514847 8166 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:55.515473 8166 tablet.cc:1620] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:55.520897 8165 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:55.522131 8165 tablet.cc:1620] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:55.524816 2002 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]). Expected version 1 got 0
I20240501 14:00:55.533106 2002 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]). Expected version 1 got 0
I20240501 14:00:55.534866 2000 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]). Expected version 1 got 0
I20240501 14:00:55.536729 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b] alter complete (version 1)
I20240501 14:00:55.542788 8168 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:55.543424 8168 tablet.cc:1620] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:55.544337 8097 raft_consensus.cc:1050] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:00:55.544979 8097 consensus_queue.cc:237] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:55.545223 2000 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b]). Expected version 1 got 0
I20240501 14:00:55.547551 5033 raft_consensus.cc:1261] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:55.548080 7865 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:55.550057 2537 raft_consensus.cc:1261] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:55.550843 2274 raft_consensus.cc:1261] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:00:55.551705 7865 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:55.552989 7279 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:00:55.555954 7279 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:55.556787 2537 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:55.561391 5033 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:55.562179 2000 catalog_manager.cc:5484] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 4 to 5, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:55.568462 2406 consensus_queue.cc:237] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:00:55.571200 2537 raft_consensus.cc:1261] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:55.572921 8097 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:55.575204 5031 raft_consensus.cc:1261] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:00:55.575866 8097 consensus_queue.cc:1035] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:00:55.578749 7279 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:55.580461 5031 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:55.582612 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01096bd6f2bf4663b70043f61ecdea5b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:55.583503 2538 raft_consensus.cc:2932] T 01096bd6f2bf4663b70043f61ecdea5b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:00:55.584931 2005 catalog_manager.cc:5484] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 5 to 6, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:55.588847 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 01096bd6f2bf4663b70043f61ecdea5b with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:55.589571 8175 tablet_replica.cc:330] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:55.589916 8175 raft_consensus.cc:2229] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:55.590261 8175 pending_rounds.cc:62] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Trying to abort 1 pending ops.
I20240501 14:00:55.590327 8175 pending_rounds.cc:68] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022887139510562816 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "01096bd6f2bf4663b70043f61ecdea5b" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } } }
I20240501 14:00:55.590718 8175 raft_consensus.cc:2894] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:00:55.591197 8175 raft_consensus.cc:2866] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:00:55.591326 8175 raft_consensus.cc:2258] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:55.592267 8175 ts_tablet_manager.cc:1875] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:55.597705 8175 ts_tablet_manager.cc:1888] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:00:55.597851 8175 log.cc:1196] T 01096bd6f2bf4663b70043f61ecdea5b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/01096bd6f2bf4663b70043f61ecdea5b
I20240501 14:00:55.598479 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 01096bd6f2bf4663b70043f61ecdea5b (table rebalance_test_table_19 [id=6094416e88404125818dcf1e3694ab85]) successfully deleted
I20240501 14:00:55.640014 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:55.649055 8166 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:55.649567 8166 tablet.cc:1620] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:55.655691 8168 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:55.656211 8168 tablet.cc:1620] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:55.656867 8165 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:55.657325 8165 tablet.cc:1620] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:55.660125 1999 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=244af303383d4ad58fddde95a10fcb3b] alter complete (version 2)
I20240501 14:00:55.711618 7695 raft_consensus.cc:482] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:55.711772 7695 raft_consensus.cc:504] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.712420 7695 leader_election.cc:288] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:55.712855 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc808ec4077a42d186aba7d4af5ac158" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:55.713012 2406 raft_consensus.cc:2454] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:55.713371 5097 leader_election.cc:302] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:55.713636 7695 raft_consensus.cc:2781] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:55.713724 7695 raft_consensus.cc:482] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:55.713784 7695 raft_consensus.cc:3037] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.715049 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc808ec4077a42d186aba7d4af5ac158" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:55.715256 2274 raft_consensus.cc:2454] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:55.715893 7695 raft_consensus.cc:504] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.716477 7695 leader_election.cc:288] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:55.716892 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc808ec4077a42d186aba7d4af5ac158" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:55.717072 2406 raft_consensus.cc:3037] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.717094 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc808ec4077a42d186aba7d4af5ac158" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:55.717278 2274 raft_consensus.cc:3037] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:55.718917 2406 raft_consensus.cc:2454] T bc808ec4077a42d186aba7d4af5ac158 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:55.719195 2274 raft_consensus.cc:2454] T bc808ec4077a42d186aba7d4af5ac158 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:55.719306 5097 leader_election.cc:302] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:55.719583 7695 raft_consensus.cc:2781] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:55.719771 7695 raft_consensus.cc:686] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:55.720119 7695 consensus_queue.cc:237] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:00:55.724428 1999 catalog_manager.cc:5484] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:55.766295 1999 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:55.776582 8165 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:55.777400 8168 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:55.785949 8166 tablet.cc:1722] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:55.794792 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=244af303383d4ad58fddde95a10fcb3b] alter complete (version 3)
I20240501 14:00:55.880210 8154 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.886438 7388 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.898783 2005 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:55.899035 2005 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:55.900161 8174 consensus_queue.cc:1035] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.902858 2005 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 7dac68f3bd2a4608ba106973c8d55de1
I20240501 14:00:55.903641 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dac68f3bd2a4608ba106973c8d55de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:55 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:55.903723 5012 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dac68f3bd2a4608ba106973c8d55de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:55 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:55.904243 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dac68f3bd2a4608ba106973c8d55de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:55 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:55.913456 8222 tablet_replica.cc:330] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:55.913709 8222 raft_consensus.cc:2229] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:55.917039 8222 raft_consensus.cc:2258] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:55.918889 8222 ts_tablet_manager.cc:1875] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:55.920244 8220 tablet_replica.cc:330] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:55.921123 8220 raft_consensus.cc:2229] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:55.921984 8220 raft_consensus.cc:2258] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:55.923771 8220 ts_tablet_manager.cc:1875] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:55.937295 8100 consensus_queue.cc:1035] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:55.940958 8221 tablet_replica.cc:330] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:55.943130 8221 raft_consensus.cc:2229] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:55.943464 8221 raft_consensus.cc:2258] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:55.944370 8222 ts_tablet_manager.cc:1888] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:55.944597 8221 ts_tablet_manager.cc:1875] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:55.947235 8220 ts_tablet_manager.cc:1888] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:55.947470 8222 log.cc:1196] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/7dac68f3bd2a4608ba106973c8d55de1
I20240501 14:00:55.948174 8222 ts_tablet_manager.cc:1909] T 7dac68f3bd2a4608ba106973c8d55de1 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:55.949415 8220 log.cc:1196] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/7dac68f3bd2a4608ba106973c8d55de1
I20240501 14:00:55.949487 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table_new_name [id=244af303383d4ad58fddde95a10fcb3b]) successfully deleted
I20240501 14:00:55.949915 8220 ts_tablet_manager.cc:1909] T 7dac68f3bd2a4608ba106973c8d55de1 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:55.950986 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table_new_name [id=244af303383d4ad58fddde95a10fcb3b]) successfully deleted
I20240501 14:00:55.958949 8221 ts_tablet_manager.cc:1888] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:55.959089 8221 log.cc:1196] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/7dac68f3bd2a4608ba106973c8d55de1
I20240501 14:00:55.959448 8221 ts_tablet_manager.cc:1909] T 7dac68f3bd2a4608ba106973c8d55de1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:55.960640 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 7dac68f3bd2a4608ba106973c8d55de1 (table rebalancer_dynamic_table_new_name [id=244af303383d4ad58fddde95a10fcb3b]) successfully deleted
I20240501 14:00:56.010490 1999 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:56.020242 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 21a6c8c55e72485c99803de34efab325 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e]), partition=
I20240501 14:00:56.020635 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a6c8c55e72485c99803de34efab325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.020709 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 21a6c8c55e72485c99803de34efab325 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e]), partition=
I20240501 14:00:56.020695 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 21a6c8c55e72485c99803de34efab325 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e]), partition=
I20240501 14:00:56.021005 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a6c8c55e72485c99803de34efab325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.021039 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a6c8c55e72485c99803de34efab325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.041118 8238 tablet_bootstrap.cc:492] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:56.042045 8236 tablet_bootstrap.cc:492] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:56.043766 8236 tablet_bootstrap.cc:654] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.044466 8238 tablet_bootstrap.cc:654] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.046316 8237 tablet_bootstrap.cc:492] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:56.052275 8237 tablet_bootstrap.cc:654] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.053969 8238 tablet_bootstrap.cc:492] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:56.054561 8238 ts_tablet_manager.cc:1367] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 14:00:56.054867 8236 tablet_bootstrap.cc:492] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:56.055032 8236 ts_tablet_manager.cc:1367] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 14:00:56.055641 8238 raft_consensus.cc:348] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.055965 8238 raft_consensus.cc:374] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.055786 8236 raft_consensus.cc:348] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.056064 8236 raft_consensus.cc:374] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.056218 8236 raft_consensus.cc:729] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.056460 8236 consensus_queue.cc:260] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.056594 8238 raft_consensus.cc:729] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.056942 8238 consensus_queue.cc:260] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.057548 8238 ts_tablet_manager.cc:1398] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:56.059718 8237 tablet_bootstrap.cc:492] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:56.059927 8237 ts_tablet_manager.cc:1367] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 14:00:56.060828 8237 raft_consensus.cc:348] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.061110 8237 raft_consensus.cc:374] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.061251 8237 raft_consensus.cc:729] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.061851 8237 consensus_queue.cc:260] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.067332 8237 ts_tablet_manager.cc:1398] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:56.071616 8236 ts_tablet_manager.cc:1398] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:00:56.144400 8154 raft_consensus.cc:482] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.144572 8154 raft_consensus.cc:504] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.145244 8154 leader_election.cc:288] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:56.146790 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a6c8c55e72485c99803de34efab325" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:56.146847 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a6c8c55e72485c99803de34efab325" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:56.147073 5033 raft_consensus.cc:2454] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:56.147102 5296 raft_consensus.cc:2454] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:56.147812 2473 leader_election.cc:302] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:56.154811 7809 consensus_queue.cc:1035] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.159914 8154 raft_consensus.cc:2781] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:56.160048 8154 raft_consensus.cc:482] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:56.160120 8154 raft_consensus.cc:3037] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.166195 8154 raft_consensus.cc:504] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.167521 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a6c8c55e72485c99803de34efab325" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:56.167800 5031 raft_consensus.cc:3037] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.171058 5031 raft_consensus.cc:2454] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:56.172575 2473 leader_election.cc:302] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:56.172899 7388 raft_consensus.cc:2781] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:56.173012 7388 raft_consensus.cc:686] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:56.173259 7388 consensus_queue.cc:237] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:56.187305 8154 leader_election.cc:288] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:56.187569 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a6c8c55e72485c99803de34efab325" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:56.187803 5296 raft_consensus.cc:3037] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.192095 5296 raft_consensus.cc:2454] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:56.232292 7809 consensus_queue.cc:1035] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.240408 2000 catalog_manager.cc:5484] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:56.324749 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:56.348148 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 95d15ba032f64cfb8422446df83094b8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.348748 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d15ba032f64cfb8422446df83094b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.350883 5012 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:56.355057 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa75617adf64020af85f4cf2bd13917 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.355598 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa75617adf64020af85f4cf2bd13917. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.356956 4879 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:00:56.361043 2518 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:00:56.361380 2388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:00:56.364997 8236 tablet_bootstrap.cc:492] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:56.365957 8237 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:56.367030 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 95d15ba032f64cfb8422446df83094b8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.367328 8236 tablet_bootstrap.cc:654] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.367455 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d15ba032f64cfb8422446df83094b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.367558 8237 tablet_bootstrap.cc:654] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.389308 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 95d15ba032f64cfb8422446df83094b8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.389803 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d15ba032f64cfb8422446df83094b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.390165 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe831b65694e75b466ec2abb082590 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.390668 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe831b65694e75b466ec2abb082590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.392835 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe831b65694e75b466ec2abb082590 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.393328 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe831b65694e75b466ec2abb082590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.393714 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe831b65694e75b466ec2abb082590 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.394107 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe831b65694e75b466ec2abb082590. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.398950 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa75617adf64020af85f4cf2bd13917 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.399212 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa75617adf64020af85f4cf2bd13917 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:56.399369 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa75617adf64020af85f4cf2bd13917. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.399616 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa75617adf64020af85f4cf2bd13917. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:56.402783 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:56.408708 8238 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:56.410104 8238 tablet_bootstrap.cc:654] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
W20240501 14:00:56.418639 4967 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.7:41601, user_credentials={real_user=slave}} blocked reactor thread for 47686.2us
I20240501 14:00:56.423084 8236 tablet_bootstrap.cc:492] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:56.423297 8236 ts_tablet_manager.cc:1367] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20240501 14:00:56.424286 8236 raft_consensus.cc:348] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.424526 8236 raft_consensus.cc:374] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.424598 8236 raft_consensus.cc:729] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.424876 8236 consensus_queue.cc:260] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.426304 8236 ts_tablet_manager.cc:1398] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:56.445235 2258 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:00:56.447751 8237 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:56.447959 8237 ts_tablet_manager.cc:1367] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.000s
I20240501 14:00:56.449942 8237 raft_consensus.cc:348] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.450223 8237 raft_consensus.cc:374] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.450317 8237 raft_consensus.cc:729] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.450618 8237 consensus_queue.cc:260] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.451182 8237 ts_tablet_manager.cc:1398] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:00:56.469660 5142 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:00:56.478636 8238 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:56.478883 8238 ts_tablet_manager.cc:1367] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.003s
I20240501 14:00:56.479598 8238 raft_consensus.cc:348] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.479882 8238 raft_consensus.cc:374] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.479962 8238 raft_consensus.cc:729] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.480226 8238 consensus_queue.cc:260] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.480804 8238 ts_tablet_manager.cc:1398] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:56.485306 8291 tablet_bootstrap.cc:492] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:56.488003 8288 tablet_bootstrap.cc:492] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:56.489873 8291 tablet_bootstrap.cc:654] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.494925 8288 tablet_bootstrap.cc:654] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.495864 8287 tablet_bootstrap.cc:492] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:56.496335 5032 raft_consensus.cc:1261] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:56.496899 8154 consensus_queue.cc:1035] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.496501 8290 tablet_bootstrap.cc:492] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:56.499861 5296 raft_consensus.cc:1261] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:56.500149 8290 tablet_bootstrap.cc:654] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.500312 5275 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:00:56.500841 8154 consensus_queue.cc:1035] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.508278 8291 tablet_bootstrap.cc:492] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:56.508466 8291 ts_tablet_manager.cc:1367] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240501 14:00:56.509311 8291 raft_consensus.cc:348] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.509552 8291 raft_consensus.cc:374] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.509632 8291 raft_consensus.cc:729] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.509845 8291 consensus_queue.cc:260] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.513923 8287 tablet_bootstrap.cc:654] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.518054 2125 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:00:56.519915 8291 ts_tablet_manager.cc:1398] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:56.520730 8291 tablet_bootstrap.cc:492] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:56.522562 8291 tablet_bootstrap.cc:654] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.530869 8288 tablet_bootstrap.cc:492] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:56.532229 8288 ts_tablet_manager.cc:1367] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.000s
I20240501 14:00:56.534119 8288 raft_consensus.cc:348] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.537429 8291 tablet_bootstrap.cc:492] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:56.537758 8291 ts_tablet_manager.cc:1367] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:00:56.537887 8288 raft_consensus.cc:374] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.538005 8288 raft_consensus.cc:729] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.538933 8291 raft_consensus.cc:348] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.539422 8288 consensus_queue.cc:260] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.540119 8288 ts_tablet_manager.cc:1398] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:56.541164 8287 tablet_bootstrap.cc:492] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:56.541366 8287 ts_tablet_manager.cc:1367] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.004s
I20240501 14:00:56.542758 8288 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:56.542465 8287 raft_consensus.cc:348] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.543421 8291 raft_consensus.cc:374] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.544417 8287 raft_consensus.cc:374] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.544590 8291 raft_consensus.cc:729] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.544768 8287 raft_consensus.cc:729] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.545370 8291 consensus_queue.cc:260] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.545845 8287 consensus_queue.cc:260] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.547297 8288 tablet_bootstrap.cc:654] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:56.548081 8287 ts_tablet_manager.cc:1398] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:56.548714 8291 ts_tablet_manager.cc:1398] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:00:56.557806 7809 raft_consensus.cc:482] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.557997 7809 raft_consensus.cc:504] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.558701 7809 leader_election.cc:288] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:56.565137 5294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d15ba032f64cfb8422446df83094b8" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:56.565459 5294 raft_consensus.cc:2454] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:56.566301 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d15ba032f64cfb8422446df83094b8" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:56.566529 2406 raft_consensus.cc:2454] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:00:56.567028 8290 tablet_bootstrap.cc:492] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:56.567059 5097 leader_election.cc:302] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:56.567320 8290 ts_tablet_manager.cc:1367] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20240501 14:00:56.567449 7809 raft_consensus.cc:2781] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:56.567569 7809 raft_consensus.cc:482] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:56.567667 7809 raft_consensus.cc:3037] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.568408 8290 raft_consensus.cc:348] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.568692 8290 raft_consensus.cc:374] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.568814 8290 raft_consensus.cc:729] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.569181 8290 consensus_queue.cc:260] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.570712 7809 raft_consensus.cc:504] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.571569 8290 ts_tablet_manager.cc:1398] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:56.572116 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d15ba032f64cfb8422446df83094b8" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:56.572314 2406 raft_consensus.cc:3037] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.575748 2406 raft_consensus.cc:2454] T 95d15ba032f64cfb8422446df83094b8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:56.576352 5097 leader_election.cc:302] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:56.577194 7872 raft_consensus.cc:2781] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:56.577360 7872 raft_consensus.cc:686] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:00:56.577616 7872 consensus_queue.cc:237] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.581768 7809 leader_election.cc:288] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:56.584714 5294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d15ba032f64cfb8422446df83094b8" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:56.584978 5294 raft_consensus.cc:3037] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.583628 2002 catalog_manager.cc:5484] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:56.587862 5294 raft_consensus.cc:2454] T 95d15ba032f64cfb8422446df83094b8 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:00:56.591593 2002 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 21a6c8c55e72485c99803de34efab325 (table rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e]). Expected version 1 got 0
I20240501 14:00:56.593133 8125 raft_consensus.cc:482] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.593318 8125 raft_consensus.cc:504] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.594125 8125 leader_election.cc:288] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:56.594946 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe831b65694e75b466ec2abb082590" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:56.595167 5161 raft_consensus.cc:2454] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:56.595580 2080 leader_election.cc:302] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:56.596161 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe831b65694e75b466ec2abb082590" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:56.596419 2274 raft_consensus.cc:2454] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:56.597007 8125 raft_consensus.cc:2781] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:56.597091 8125 raft_consensus.cc:482] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:56.597168 8125 raft_consensus.cc:3037] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.601449 8125 raft_consensus.cc:504] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.602809 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe831b65694e75b466ec2abb082590" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:56.603068 2274 raft_consensus.cc:3037] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.608671 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe831b65694e75b466ec2abb082590" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:56.608913 5161 raft_consensus.cc:3037] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.609395 8125 leader_election.cc:288] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:56.617125 2274 raft_consensus.cc:2454] T 3efe831b65694e75b466ec2abb082590 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:56.617130 8305 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:56.617643 2080 leader_election.cc:302] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:56.617975 8125 raft_consensus.cc:2781] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:56.622740 2000 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 21a6c8c55e72485c99803de34efab325 (table rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e]). Expected version 1 got 0
I20240501 14:00:56.629462 2005 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 21a6c8c55e72485c99803de34efab325 (table rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e]). Expected version 1 got 0
I20240501 14:00:56.634452 5161 raft_consensus.cc:2454] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:56.636044 7388 raft_consensus.cc:482] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.636204 7388 raft_consensus.cc:504] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.636942 7388 leader_election.cc:288] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:56.637635 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa75617adf64020af85f4cf2bd13917" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
W20240501 14:00:56.638227 2471 leader_election.cc:341] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:56.642736 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa75617adf64020af85f4cf2bd13917" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:56.642993 5032 raft_consensus.cc:2454] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:56.643505 2473 leader_election.cc:302] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters: d1a414d381f44829a2510cb76d481887
I20240501 14:00:56.643932 7388 raft_consensus.cc:2781] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:56.644006 7388 raft_consensus.cc:482] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:56.644073 7388 raft_consensus.cc:3037] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.648855 7388 raft_consensus.cc:504] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.650202 7388 leader_election.cc:288] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:56.650362 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa75617adf64020af85f4cf2bd13917" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:56.650616 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa75617adf64020af85f4cf2bd13917" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:56.650842 5032 raft_consensus.cc:3037] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:00:56.651288 2471 leader_election.cc:341] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:56.653676 8125 raft_consensus.cc:686] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:56.654042 5032 raft_consensus.cc:2454] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:56.653978 8125 consensus_queue.cc:237] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:56.654542 2473 leader_election.cc:302] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters: d1a414d381f44829a2510cb76d481887
I20240501 14:00:56.655011 7388 raft_consensus.cc:2781] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:56.655457 7388 raft_consensus.cc:686] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:56.655719 7388 consensus_queue.cc:237] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.656812 8288 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:56.656950 8288 ts_tablet_manager.cc:1367] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.019s
I20240501 14:00:56.661564 8288 raft_consensus.cc:348] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.617779 8305 tablet.cc:1620] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:56.664319 8288 raft_consensus.cc:374] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:56.664404 8288 raft_consensus.cc:729] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:56.664625 8288 consensus_queue.cc:260] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.663240 8304 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:56.665359 8288 ts_tablet_manager.cc:1398] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240501 14:00:56.665480 2002 catalog_manager.cc:5484] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:56.665728 8304 tablet.cc:1620] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:56.683518 8310 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:56.684156 8310 tablet.cc:1620] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:56.708375 2002 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 21a6c8c55e72485c99803de34efab325 (table rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e]). Expected version 1 got 0
I20240501 14:00:56.716022 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e] alter complete (version 1)
I20240501 14:00:56.731668 2002 catalog_manager.cc:5484] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:56.840056 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:56.851837 8304 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:56.852371 8304 tablet.cc:1620] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:56.853968 8310 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:56.854482 8310 tablet.cc:1620] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:56.857697 8306 raft_consensus.cc:482] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:56.857849 8306 raft_consensus.cc:504] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:56.858569 8306 leader_election.cc:288] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:56.859236 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa75617adf64020af85f4cf2bd13917" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:56.860414 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa75617adf64020af85f4cf2bd13917" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:56.860671 5031 raft_consensus.cc:2379] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1a414d381f44829a2510cb76d481887 in current term 1: Already voted for candidate 745ac12076e84effa5bde1ecbd6a32ec in this term.
I20240501 14:00:56.860888 8305 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:56.861373 8305 tablet.cc:1620] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:00:56.862054 2082 leader_election.cc:302] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:00:56.870733 8125 raft_consensus.cc:3037] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:56.873664 8125 raft_consensus.cc:2726] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:56.878544 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=08bd3f4ee2624b0b9059e55d5a7e899e] alter complete (version 2)
I20240501 14:00:56.974085 7695 consensus_queue.cc:1035] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.988584 7695 consensus_queue.cc:1035] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:56.989609 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:57.002007 8304 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.004160 8310 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.006958 8305 tablet.cc:1722] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.019522 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=08bd3f4ee2624b0b9059e55d5a7e899e] alter complete (version 3)
I20240501 14:00:57.091132 8307 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.096632 8307 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.106948 8306 consensus_queue.cc:1035] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.111858 8322 consensus_queue.cc:1035] T 3efe831b65694e75b466ec2abb082590 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.124166 2005 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:57.124382 2005 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:57.127614 2005 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 21a6c8c55e72485c99803de34efab325
I20240501 14:00:57.128439 5012 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a6c8c55e72485c99803de34efab325 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:57 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:00:57.128743 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a6c8c55e72485c99803de34efab325 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:57 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:57.129408 8326 tablet_replica.cc:330] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:57.129798 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a6c8c55e72485c99803de34efab325 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:57 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:57.129727 8326 raft_consensus.cc:2229] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.131040 8326 raft_consensus.cc:2258] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.134799 8326 ts_tablet_manager.cc:1875] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:57.144374 8327 tablet_replica.cc:330] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:57.144428 8326 ts_tablet_manager.cc:1888] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:57.144619 8326 log.cc:1196] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/21a6c8c55e72485c99803de34efab325
I20240501 14:00:57.144848 8325 tablet_replica.cc:330] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:00:57.144866 8327 raft_consensus.cc:2229] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:57.145215 8326 ts_tablet_manager.cc:1909] T 21a6c8c55e72485c99803de34efab325 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:57.145267 8327 raft_consensus.cc:2258] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.145229 8325 raft_consensus.cc:2229] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.145907 8325 raft_consensus.cc:2258] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.146476 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 21a6c8c55e72485c99803de34efab325 (table rebalancer_dynamic_table_new_name [id=08bd3f4ee2624b0b9059e55d5a7e899e]) successfully deleted
I20240501 14:00:57.146852 8325 ts_tablet_manager.cc:1875] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:57.146873 8327 ts_tablet_manager.cc:1875] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:57.152757 8325 ts_tablet_manager.cc:1888] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:57.152907 8325 log.cc:1196] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/21a6c8c55e72485c99803de34efab325
I20240501 14:00:57.153187 8327 ts_tablet_manager.cc:1888] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:57.153314 8327 log.cc:1196] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/21a6c8c55e72485c99803de34efab325
I20240501 14:00:57.153424 8325 ts_tablet_manager.cc:1909] T 21a6c8c55e72485c99803de34efab325 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:00:57.153657 8327 ts_tablet_manager.cc:1909] T 21a6c8c55e72485c99803de34efab325 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:57.154606 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 21a6c8c55e72485c99803de34efab325 (table rebalancer_dynamic_table_new_name [id=08bd3f4ee2624b0b9059e55d5a7e899e]) successfully deleted
I20240501 14:00:57.154752 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 21a6c8c55e72485c99803de34efab325 (table rebalancer_dynamic_table_new_name [id=08bd3f4ee2624b0b9059e55d5a7e899e]) successfully deleted
I20240501 14:00:57.235472 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:57.242973 5275 tablet_service.cc:1467] Processing CreateTablet for tablet faad897799c04983b6a0bf1daad866c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7]), partition=
I20240501 14:00:57.243273 2257 tablet_service.cc:1467] Processing CreateTablet for tablet faad897799c04983b6a0bf1daad866c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7]), partition=
I20240501 14:00:57.243417 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faad897799c04983b6a0bf1daad866c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.243611 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faad897799c04983b6a0bf1daad866c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.244980 2518 tablet_service.cc:1467] Processing CreateTablet for tablet faad897799c04983b6a0bf1daad866c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7]), partition=
I20240501 14:00:57.245327 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faad897799c04983b6a0bf1daad866c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.252588 8330 tablet_bootstrap.cc:492] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:57.254415 8330 tablet_bootstrap.cc:654] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.257819 8330 tablet_bootstrap.cc:492] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:57.258054 8330 ts_tablet_manager.cc:1367] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:00:57.258919 8330 raft_consensus.cc:348] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.258867 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:57.260747 8329 tablet_bootstrap.cc:492] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:57.261168 8330 raft_consensus.cc:374] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.261287 8330 raft_consensus.cc:729] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.261559 8330 consensus_queue.cc:260] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.263721 8329 tablet_bootstrap.cc:654] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.263760 8330 ts_tablet_manager.cc:1398] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:57.268810 8329 tablet_bootstrap.cc:492] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:57.269039 8329 ts_tablet_manager.cc:1367] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 14:00:57.269775 8329 raft_consensus.cc:348] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.270066 8329 raft_consensus.cc:374] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.270175 8329 raft_consensus.cc:729] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.270453 8329 consensus_queue.cc:260] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.271052 8329 ts_tablet_manager.cc:1398] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:57.271476 8333 tablet_bootstrap.cc:492] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:57.273321 8333 tablet_bootstrap.cc:654] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.280761 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 527b1cd4c73749f39b8659312df939ff (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.281281 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527b1cd4c73749f39b8659312df939ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.281695 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 527b1cd4c73749f39b8659312df939ff (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.282068 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527b1cd4c73749f39b8659312df939ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.282177 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 741357494d3a4842935671afd1dbe677 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.282641 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 741357494d3a4842935671afd1dbe677. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.283134 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 527b1cd4c73749f39b8659312df939ff (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.283550 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527b1cd4c73749f39b8659312df939ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.285987 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 741357494d3a4842935671afd1dbe677 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.286440 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 741357494d3a4842935671afd1dbe677. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.289232 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 741357494d3a4842935671afd1dbe677 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.289665 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 741357494d3a4842935671afd1dbe677. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.295569 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 7655e45073d947eabec36e6de7d6a688 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.296046 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7655e45073d947eabec36e6de7d6a688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.299599 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 7655e45073d947eabec36e6de7d6a688 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.299669 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 7655e45073d947eabec36e6de7d6a688 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:57.300148 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7655e45073d947eabec36e6de7d6a688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.300177 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7655e45073d947eabec36e6de7d6a688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:57.302783 8335 tablet_bootstrap.cc:492] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:57.305980 8335 tablet_bootstrap.cc:654] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.313338 8338 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:57.315055 8338 tablet_bootstrap.cc:654] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.315698 8335 tablet_bootstrap.cc:492] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:57.315850 8335 ts_tablet_manager.cc:1367] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 14:00:57.316504 8335 raft_consensus.cc:348] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.316699 8335 raft_consensus.cc:374] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.317351 8329 tablet_bootstrap.cc:492] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:57.318809 8329 tablet_bootstrap.cc:654] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.318850 8335 raft_consensus.cc:729] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.319108 8335 consensus_queue.cc:260] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.320127 8330 tablet_bootstrap.cc:492] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:57.321692 8330 tablet_bootstrap.cc:654] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.322234 8336 tablet_bootstrap.cc:492] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:57.322716 8333 tablet_bootstrap.cc:492] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:57.322858 8333 ts_tablet_manager.cc:1367] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.007s
I20240501 14:00:57.323623 8333 raft_consensus.cc:348] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.323848 8333 raft_consensus.cc:374] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.323931 8333 raft_consensus.cc:729] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.324021 8336 tablet_bootstrap.cc:654] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.324170 8333 consensus_queue.cc:260] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.324692 8333 ts_tablet_manager.cc:1398] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:57.324972 8333 tablet_bootstrap.cc:492] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:57.331848 8333 tablet_bootstrap.cc:654] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.343825 8335 ts_tablet_manager.cc:1398] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240501 14:00:57.352022 8338 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:57.352069 8342 tablet_bootstrap.cc:492] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:57.352748 8338 ts_tablet_manager.cc:1367] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240501 14:00:57.353897 8333 tablet_bootstrap.cc:492] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:57.354143 8333 ts_tablet_manager.cc:1367] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.009s
I20240501 14:00:57.355223 8341 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:57.355605 8330 tablet_bootstrap.cc:492] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:57.355806 8330 ts_tablet_manager.cc:1367] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.009s
I20240501 14:00:57.356000 8333 raft_consensus.cc:348] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.356308 8333 raft_consensus.cc:374] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.356426 8333 raft_consensus.cc:729] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.356544 8330 raft_consensus.cc:348] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.356807 8330 raft_consensus.cc:374] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.356911 8330 raft_consensus.cc:729] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.357187 8330 consensus_queue.cc:260] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.358871 8342 tablet_bootstrap.cc:654] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.360014 8333 consensus_queue.cc:260] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.361769 8330 ts_tablet_manager.cc:1398] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:00:57.365742 8333 ts_tablet_manager.cc:1398] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:00:57.365897 8341 tablet_bootstrap.cc:654] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.369359 8329 tablet_bootstrap.cc:492] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:57.369521 8329 ts_tablet_manager.cc:1367] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.005s
I20240501 14:00:57.370265 8329 raft_consensus.cc:348] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.370316 8338 raft_consensus.cc:348] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.370517 8329 raft_consensus.cc:374] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.370541 8338 raft_consensus.cc:374] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.370589 8329 raft_consensus.cc:729] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.370625 8338 raft_consensus.cc:729] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.370859 8338 consensus_queue.cc:260] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.370862 8329 consensus_queue.cc:260] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.371412 8338 ts_tablet_manager.cc:1398] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:57.371563 8329 ts_tablet_manager.cc:1398] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:57.372810 8336 tablet_bootstrap.cc:492] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:57.372958 8336 ts_tablet_manager.cc:1367] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.011s
I20240501 14:00:57.373719 8336 raft_consensus.cc:348] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.373909 8336 raft_consensus.cc:374] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.373976 8336 raft_consensus.cc:729] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.374207 8336 consensus_queue.cc:260] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.374791 8336 ts_tablet_manager.cc:1398] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:57.377516 8336 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:57.379630 8336 tablet_bootstrap.cc:654] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:57.387246 8336 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:57.387476 8336 ts_tablet_manager.cc:1367] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 14:00:57.388412 8336 raft_consensus.cc:348] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.388753 8336 raft_consensus.cc:374] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.388851 8341 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:57.388885 8336 raft_consensus.cc:729] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.389364 8342 tablet_bootstrap.cc:492] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:57.389235 8336 consensus_queue.cc:260] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.389602 8342 ts_tablet_manager.cc:1367] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20240501 14:00:57.389621 8341 ts_tablet_manager.cc:1367] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.005s
I20240501 14:00:57.390444 8341 raft_consensus.cc:348] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.390704 8336 ts_tablet_manager.cc:1398] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:57.390728 8341 raft_consensus.cc:374] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.390816 8341 raft_consensus.cc:729] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.391093 8341 consensus_queue.cc:260] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.391884 8341 ts_tablet_manager.cc:1398] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:57.391245 8342 raft_consensus.cc:348] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.400068 8342 raft_consensus.cc:374] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:57.400277 8342 raft_consensus.cc:729] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:57.400610 8342 consensus_queue.cc:260] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.404300 8342 ts_tablet_manager.cc:1398] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:00:57.441010 8150 raft_consensus.cc:482] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.441215 8150 raft_consensus.cc:504] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.442095 8150 leader_election.cc:288] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:57.442598 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527b1cd4c73749f39b8659312df939ff" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:57.442730 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527b1cd4c73749f39b8659312df939ff" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:57.442849 2406 raft_consensus.cc:2454] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:57.442935 2538 raft_consensus.cc:2454] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:57.443363 4834 leader_election.cc:302] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:57.443636 8150 raft_consensus.cc:2781] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.443761 8150 raft_consensus.cc:482] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.443845 8150 raft_consensus.cc:3037] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.446286 8150 raft_consensus.cc:504] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.446950 7388 raft_consensus.cc:482] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.447094 7388 raft_consensus.cc:504] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.447585 8150 leader_election.cc:288] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:00:57.447700 7388 leader_election.cc:288] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:57.448540 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faad897799c04983b6a0bf1daad866c4" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:57.448534 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527b1cd4c73749f39b8659312df939ff" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:57.448725 2406 raft_consensus.cc:3037] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.448776 5292 raft_consensus.cc:2454] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:57.449896 2473 leader_election.cc:302] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:57.450178 7388 raft_consensus.cc:2781] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.450268 7388 raft_consensus.cc:482] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.450343 7388 raft_consensus.cc:3037] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.451376 2406 raft_consensus.cc:2454] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:57.451855 4834 leader_election.cc:302] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:57.452142 8150 raft_consensus.cc:2781] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.452286 8150 raft_consensus.cc:686] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:57.452584 8150 consensus_queue.cc:237] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:57.452952 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faad897799c04983b6a0bf1daad866c4" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:57.453150 2274 raft_consensus.cc:2454] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:57.452988 7388 raft_consensus.cc:504] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.453914 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527b1cd4c73749f39b8659312df939ff" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:57.454106 2538 raft_consensus.cc:3037] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.454360 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faad897799c04983b6a0bf1daad866c4" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:57.454464 7388 leader_election.cc:288] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:00:57.454603 5296 raft_consensus.cc:3037] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.456737 2538 raft_consensus.cc:2454] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:57.457288 5296 raft_consensus.cc:2454] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:57.457666 2473 leader_election.cc:302] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:57.458196 7388 raft_consensus.cc:2781] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.458231 8174 raft_consensus.cc:482] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.458307 7388 raft_consensus.cc:686] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:57.458367 8174 raft_consensus.cc:504] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.458575 7388 consensus_queue.cc:237] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.459110 8174 leader_election.cc:288] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:57.459878 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faad897799c04983b6a0bf1daad866c4" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:57.460057 2274 raft_consensus.cc:3037] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.460124 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7655e45073d947eabec36e6de7d6a688" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:57.460186 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7655e45073d947eabec36e6de7d6a688" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:57.460314 5296 raft_consensus.cc:2454] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:57.460414 5031 raft_consensus.cc:2454] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:57.460681 2213 leader_election.cc:302] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:57.460992 8174 raft_consensus.cc:2781] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.461068 8174 raft_consensus.cc:482] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.461122 8174 raft_consensus.cc:3037] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.462756 2002 catalog_manager.cc:5484] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:57.466606 8174 raft_consensus.cc:504] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.467255 8174 leader_election.cc:288] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:57.467499 2274 raft_consensus.cc:2454] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:57.467805 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7655e45073d947eabec36e6de7d6a688" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:57.468025 5296 raft_consensus.cc:3037] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.467975 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7655e45073d947eabec36e6de7d6a688" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:57.468190 5031 raft_consensus.cc:3037] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.471021 5031 raft_consensus.cc:2454] T 7655e45073d947eabec36e6de7d6a688 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:57.471882 5296 raft_consensus.cc:2454] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:57.472342 2213 leader_election.cc:302] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:57.473455 8174 raft_consensus.cc:2781] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.473582 8174 raft_consensus.cc:686] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:57.473821 8174 consensus_queue.cc:237] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:57.476178 2002 catalog_manager.cc:5484] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:57.502188 2002 catalog_manager.cc:5484] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:57.612980 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:57.619453 2274 raft_consensus.cc:1261] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:57.619911 5296 raft_consensus.cc:1261] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:57.619997 8307 consensus_queue.cc:1035] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.620998 8307 consensus_queue.cc:1035] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.623631 8357 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:57.624614 8357 tablet.cc:1620] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:57.625409 2002 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet faad897799c04983b6a0bf1daad866c4 (table rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7]). Expected version 1 got 0
I20240501 14:00:57.632340 8359 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:57.633033 8359 tablet.cc:1620] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:57.637472 2002 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet faad897799c04983b6a0bf1daad866c4 (table rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7]). Expected version 1 got 0
I20240501 14:00:57.638325 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7] alter complete (version 1)
I20240501 14:00:57.658907 2000 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet faad897799c04983b6a0bf1daad866c4 (table rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7]). Expected version 1 got 0
I20240501 14:00:57.661844 8360 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:57.662591 8360 tablet.cc:1620] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:57.663918 2002 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet faad897799c04983b6a0bf1daad866c4 (table rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7]). Expected version 1 got 0
I20240501 14:00:57.716259 8150 raft_consensus.cc:482] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:57.716457 8150 raft_consensus.cc:504] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.717165 8150 leader_election.cc:288] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:57.717734 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741357494d3a4842935671afd1dbe677" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:57.717968 5164 raft_consensus.cc:2454] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:57.718297 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741357494d3a4842935671afd1dbe677" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:57.718436 4834 leader_election.cc:302] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:57.718539 2143 raft_consensus.cc:2454] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:57.718737 8150 raft_consensus.cc:2781] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:57.718823 8150 raft_consensus.cc:482] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:57.718931 8150 raft_consensus.cc:3037] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.720885 8150 raft_consensus.cc:504] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.721467 8150 leader_election.cc:288] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:57.721832 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741357494d3a4842935671afd1dbe677" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:57.722086 5164 raft_consensus.cc:3037] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.722051 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741357494d3a4842935671afd1dbe677" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:57.722231 2143 raft_consensus.cc:3037] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:57.724905 2143 raft_consensus.cc:2454] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:57.724920 5164 raft_consensus.cc:2454] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:57.725360 4834 leader_election.cc:302] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:57.725661 8150 raft_consensus.cc:2781] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:57.725818 8150 raft_consensus.cc:686] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:57.726092 8150 consensus_queue.cc:237] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:57.729890 2002 catalog_manager.cc:5484] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:57.749028 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:57.757128 8357 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:57.757650 8357 tablet.cc:1620] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:00:57.757771 8359 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:57.758230 8359 tablet.cc:1620] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:00:57.758949 8360 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:57.759385 8360 tablet.cc:1620] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:57.771085 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=757f3da0c3ae4801923e0ee549be62e7] alter complete (version 2)
I20240501 14:00:57.844738 8150 consensus_queue.cc:1035] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.852185 7862 consensus_queue.cc:1035] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.856585 8150 consensus_queue.cc:1035] T 527b1cd4c73749f39b8659312df939ff P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.859447 7862 consensus_queue.cc:1035] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:57.874387 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:57.888959 8357 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.889712 8359 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.908190 8360 tablet.cc:1722] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:57.907986 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=757f3da0c3ae4801923e0ee549be62e7] alter complete (version 3)
I20240501 14:00:58.012508 2003 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:58.012791 2003 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:58.016104 2003 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet faad897799c04983b6a0bf1daad866c4
I20240501 14:00:58.016858 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet faad897799c04983b6a0bf1daad866c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:58.017287 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet faad897799c04983b6a0bf1daad866c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:00:58.017443 8372 tablet_replica.cc:330] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:58.017695 8372 raft_consensus.cc:2229] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:58.017939 8372 raft_consensus.cc:2258] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.017976 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet faad897799c04983b6a0bf1daad866c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:00:58.018586 8374 tablet_replica.cc:330] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:00:58.018796 8374 raft_consensus.cc:2229] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:58.019122 8374 raft_consensus.cc:2258] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.019443 8372 ts_tablet_manager.cc:1875] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.025416 8374 ts_tablet_manager.cc:1875] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.026286 8372 ts_tablet_manager.cc:1888] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.026466 8372 log.cc:1196] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/faad897799c04983b6a0bf1daad866c4
I20240501 14:00:58.026942 8372 ts_tablet_manager.cc:1909] T faad897799c04983b6a0bf1daad866c4 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:58.028019 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet faad897799c04983b6a0bf1daad866c4 (table rebalancer_dynamic_table_new_name [id=757f3da0c3ae4801923e0ee549be62e7]) successfully deleted
I20240501 14:00:58.030843 8373 tablet_replica.cc:330] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:00:58.031067 8373 raft_consensus.cc:2229] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:58.031262 8374 ts_tablet_manager.cc:1888] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.031306 8373 raft_consensus.cc:2258] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.031394 8374 log.cc:1196] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/faad897799c04983b6a0bf1daad866c4
I20240501 14:00:58.031745 8374 ts_tablet_manager.cc:1909] T faad897799c04983b6a0bf1daad866c4 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:00:58.032264 8373 ts_tablet_manager.cc:1875] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.032882 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet faad897799c04983b6a0bf1daad866c4 (table rebalancer_dynamic_table_new_name [id=757f3da0c3ae4801923e0ee549be62e7]) successfully deleted
I20240501 14:00:58.036619 8373 ts_tablet_manager.cc:1888] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.036732 8373 log.cc:1196] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/faad897799c04983b6a0bf1daad866c4
I20240501 14:00:58.036952 8373 ts_tablet_manager.cc:1909] T faad897799c04983b6a0bf1daad866c4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:00:58.037933 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet faad897799c04983b6a0bf1daad866c4 (table rebalancer_dynamic_table_new_name [id=757f3da0c3ae4801923e0ee549be62e7]) successfully deleted
I20240501 14:00:58.118551 2003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:58.126760 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 580da5dc03db41d89ff3b7a4a97927b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6ac5c5b725641afa22a7b0464219091]), partition=
I20240501 14:00:58.127164 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 580da5dc03db41d89ff3b7a4a97927b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6ac5c5b725641afa22a7b0464219091]), partition=
I20240501 14:00:58.127202 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580da5dc03db41d89ff3b7a4a97927b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.127493 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580da5dc03db41d89ff3b7a4a97927b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.131527 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 580da5dc03db41d89ff3b7a4a97927b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d6ac5c5b725641afa22a7b0464219091]), partition=
I20240501 14:00:58.131911 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580da5dc03db41d89ff3b7a4a97927b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.142679 8375 tablet_bootstrap.cc:492] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:58.144467 8375 tablet_bootstrap.cc:654] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.156257 8375 tablet_bootstrap.cc:492] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:58.156450 8375 ts_tablet_manager.cc:1367] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20240501 14:00:58.157271 8375 raft_consensus.cc:348] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.157516 8375 raft_consensus.cc:374] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.157593 8375 raft_consensus.cc:729] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.157824 8375 consensus_queue.cc:260] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.158378 8375 ts_tablet_manager.cc:1398] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:58.158699 8379 tablet_bootstrap.cc:492] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:58.160395 8379 tablet_bootstrap.cc:654] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.175029 8381 tablet_bootstrap.cc:492] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:58.176672 8381 tablet_bootstrap.cc:654] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.178675 8379 tablet_bootstrap.cc:492] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:58.178896 8379 ts_tablet_manager.cc:1367] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.016s
I20240501 14:00:58.179023 8381 tablet_bootstrap.cc:492] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:58.179196 8381 ts_tablet_manager.cc:1367] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:58.179837 8379 raft_consensus.cc:348] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.180147 8379 raft_consensus.cc:374] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.180073 8381 raft_consensus.cc:348] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.180241 8379 raft_consensus.cc:729] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.180369 8381 raft_consensus.cc:374] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.180485 8381 raft_consensus.cc:729] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.180495 8379 consensus_queue.cc:260] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.180770 8381 consensus_queue.cc:260] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.181387 8381 ts_tablet_manager.cc:1398] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:00:58.185140 8364 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:58.221688 2082 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.8:34221, user_credentials={real_user=slave}} blocked reactor thread for 74010us
W20240501 14:00:58.230271 4967 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.7:41601, user_credentials={real_user=slave}} blocked reactor thread for 93625.3us
I20240501 14:00:58.235673 8379 ts_tablet_manager.cc:1398] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.056s user 0.015s sys 0.039s
I20240501 14:00:58.241446 8328 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.272430 8150 raft_consensus.cc:482] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:58.272599 8150 raft_consensus.cc:504] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.273289 8150 leader_election.cc:288] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:58.273878 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580da5dc03db41d89ff3b7a4a97927b7" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:58.274137 5164 raft_consensus.cc:2454] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:58.274613 4834 leader_election.cc:302] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:58.274926 8150 raft_consensus.cc:2781] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:58.274904 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580da5dc03db41d89ff3b7a4a97927b7" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:58.275060 8150 raft_consensus.cc:482] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:58.275143 8150 raft_consensus.cc:3037] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.275192 2143 raft_consensus.cc:2454] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:58.277484 8150 raft_consensus.cc:504] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.278250 8150 leader_election.cc:288] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:58.278558 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580da5dc03db41d89ff3b7a4a97927b7" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:58.278751 2145 raft_consensus.cc:3037] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.278781 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580da5dc03db41d89ff3b7a4a97927b7" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:58.279012 5161 raft_consensus.cc:3037] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.280987 2145 raft_consensus.cc:2454] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:58.281203 5161 raft_consensus.cc:2454] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:58.281541 4834 leader_election.cc:302] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:58.282408 8150 raft_consensus.cc:2781] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:58.282591 8150 raft_consensus.cc:686] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:58.282752 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:58.283730 8150 consensus_queue.cc:237] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:00:58.293009 2003 catalog_manager.cc:5484] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:58.293696 2518 tablet_service.cc:1467] Processing CreateTablet for tablet f647f3f27faf435f81e076bfc2ef0795 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.294188 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647f3f27faf435f81e076bfc2ef0795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.299420 2516 tablet_service.cc:1467] Processing CreateTablet for tablet e081778a0f67486a904f3bed7dfe8644 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.299872 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e081778a0f67486a904f3bed7dfe8644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.300009 5144 tablet_service.cc:1467] Processing CreateTablet for tablet f647f3f27faf435f81e076bfc2ef0795 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.299420 2517 tablet_service.cc:1467] Processing CreateTablet for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.300423 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647f3f27faf435f81e076bfc2ef0795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.300463 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.305650 8375 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:00:58.307337 8375 tablet_bootstrap.cc:654] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.307729 5275 tablet_service.cc:1467] Processing CreateTablet for tablet e081778a0f67486a904f3bed7dfe8644 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.307826 5012 tablet_service.cc:1467] Processing CreateTablet for tablet f647f3f27faf435f81e076bfc2ef0795 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.308141 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e081778a0f67486a904f3bed7dfe8644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.308243 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647f3f27faf435f81e076bfc2ef0795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.310967 5276 tablet_service.cc:1467] Processing CreateTablet for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.311432 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.314018 8395 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:58.314852 5014 tablet_service.cc:1467] Processing CreateTablet for tablet e081778a0f67486a904f3bed7dfe8644 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.314854 5013 tablet_service.cc:1467] Processing CreateTablet for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:58.315361 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e081778a0f67486a904f3bed7dfe8644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.316424 8395 tablet_bootstrap.cc:654] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.321516 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.334946 8397 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:58.340838 8397 tablet_bootstrap.cc:654] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.343148 8398 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:58.344894 8398 tablet_bootstrap.cc:654] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.344974 8395 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:58.345302 8395 ts_tablet_manager.cc:1367] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.020s
I20240501 14:00:58.345872 8375 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:00:58.346117 8375 ts_tablet_manager.cc:1367] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.001s
I20240501 14:00:58.346221 8395 raft_consensus.cc:348] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.346587 8395 raft_consensus.cc:374] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.346730 8395 raft_consensus.cc:729] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.346961 8375 raft_consensus.cc:348] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.347251 8375 raft_consensus.cc:374] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.347369 8375 raft_consensus.cc:729] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.347265 8395 consensus_queue.cc:260] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.347682 8375 consensus_queue.cc:260] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.348765 8395 ts_tablet_manager.cc:1398] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:58.349171 8397 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:58.349237 8395 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:58.349373 8397 ts_tablet_manager.cc:1367] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240501 14:00:58.351676 8395 tablet_bootstrap.cc:654] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.352887 8375 ts_tablet_manager.cc:1398] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:58.361235 8395 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:58.361539 8395 ts_tablet_manager.cc:1367] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240501 14:00:58.362596 8395 raft_consensus.cc:348] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.362923 8395 raft_consensus.cc:374] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.363041 8395 raft_consensus.cc:729] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.363400 8395 consensus_queue.cc:260] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.363442 8397 raft_consensus.cc:348] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.363662 8397 raft_consensus.cc:374] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.363759 8397 raft_consensus.cc:729] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.364239 8395 ts_tablet_manager.cc:1398] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:58.364676 8395 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:58.366374 8395 tablet_bootstrap.cc:654] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.376579 8395 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:58.376848 8395 ts_tablet_manager.cc:1367] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240501 14:00:58.377283 8398 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:58.377502 8398 ts_tablet_manager.cc:1367] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.004s
I20240501 14:00:58.377703 8395 raft_consensus.cc:348] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.378005 8395 raft_consensus.cc:374] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.378145 8395 raft_consensus.cc:729] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.378196 8397 consensus_queue.cc:260] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.378343 8398 raft_consensus.cc:348] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.378458 8395 consensus_queue.cc:260] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.378654 8398 raft_consensus.cc:374] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.378783 8398 raft_consensus.cc:729] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.379194 8395 ts_tablet_manager.cc:1398] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:58.379207 8398 consensus_queue.cc:260] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.380029 8398 ts_tablet_manager.cc:1398] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:00:58.380986 8398 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:58.382899 8398 tablet_bootstrap.cc:654] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.389132 8397 ts_tablet_manager.cc:1398] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240501 14:00:58.389518 8397 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:58.391175 8397 tablet_bootstrap.cc:654] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.391350 8398 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:58.391649 8398 ts_tablet_manager.cc:1367] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 14:00:58.392943 8398 raft_consensus.cc:348] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.394284 8398 raft_consensus.cc:374] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.394392 8398 raft_consensus.cc:729] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.394691 8398 consensus_queue.cc:260] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.394990 8397 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:58.395241 8397 ts_tablet_manager.cc:1367] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 14:00:58.395280 8398 ts_tablet_manager.cc:1398] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:58.396077 8397 raft_consensus.cc:348] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.396389 8397 raft_consensus.cc:374] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.396524 8397 raft_consensus.cc:729] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.398267 8397 consensus_queue.cc:260] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.399050 8397 ts_tablet_manager.cc:1398] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:58.399391 8397 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:58.402771 8397 tablet_bootstrap.cc:654] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.410043 8397 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:58.410226 8397 ts_tablet_manager.cc:1367] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240501 14:00:58.411042 8397 raft_consensus.cc:348] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.411311 8397 raft_consensus.cc:374] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.411397 8397 raft_consensus.cc:729] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.411624 8397 consensus_queue.cc:260] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.412078 8397 ts_tablet_manager.cc:1398] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:58.416882 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:58.423784 5161 raft_consensus.cc:1261] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:58.423878 2145 raft_consensus.cc:1261] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:58.424489 8150 consensus_queue.cc:1035] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.424729 8364 consensus_queue.cc:1035] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:58.430137 2002 catalog_manager.cc:5576] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309) does not have the latest schema for tablet 580da5dc03db41d89ff3b7a4a97927b7 (table rebalancer_dynamic_table [id=d6ac5c5b725641afa22a7b0464219091]). Expected version 1 got 0
I20240501 14:00:58.430207 8408 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:58.430956 8408 tablet.cc:1620] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:58.432173 8409 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:58.433109 8409 tablet.cc:1620] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:00:58.443543 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=d6ac5c5b725641afa22a7b0464219091] alter complete (version 1)
I20240501 14:00:58.445160 8410 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:58.445822 8410 tablet.cc:1620] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:58.446616 2003 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet 580da5dc03db41d89ff3b7a4a97927b7 (table rebalancer_dynamic_table [id=d6ac5c5b725641afa22a7b0464219091]). Expected version 1 got 0
I20240501 14:00:58.527233 8161 raft_consensus.cc:482] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:58.527415 8161 raft_consensus.cc:504] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.528136 8161 leader_election.cc:288] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:58.528632 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e71ec5dcbe4a08a0a7da5944dfe77d" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:58.528648 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e71ec5dcbe4a08a0a7da5944dfe77d" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:58.528861 5292 raft_consensus.cc:2454] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:58.528892 5032 raft_consensus.cc:2454] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:58.529259 2473 leader_election.cc:302] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:58.529508 8161 raft_consensus.cc:2781] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:58.529598 8161 raft_consensus.cc:482] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:58.529677 8161 raft_consensus.cc:3037] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.531698 8161 raft_consensus.cc:504] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.532280 8161 leader_election.cc:288] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:00:58.532815 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e71ec5dcbe4a08a0a7da5944dfe77d" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:58.532856 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e71ec5dcbe4a08a0a7da5944dfe77d" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:58.533053 5032 raft_consensus.cc:3037] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.533059 5292 raft_consensus.cc:3037] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.535398 5292 raft_consensus.cc:2454] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:58.535416 5032 raft_consensus.cc:2454] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:58.536149 2473 leader_election.cc:302] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:58.536445 8161 raft_consensus.cc:2781] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:58.536602 8161 raft_consensus.cc:686] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:58.536825 8161 consensus_queue.cc:237] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:00:58.541368 2002 catalog_manager.cc:5484] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:58.547852 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:58.556399 8408 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:58.556948 8408 tablet.cc:1620] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:00:58.557754 8409 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:58.558290 8409 tablet.cc:1620] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:00:58.561215 8410 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:58.561687 8410 tablet.cc:1620] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:00:58.568342 2003 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=d6ac5c5b725641afa22a7b0464219091] alter complete (version 2)
I20240501 14:00:58.588780 8387 raft_consensus.cc:482] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:58.588976 8387 raft_consensus.cc:504] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.589751 8387 leader_election.cc:288] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:58.590353 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647f3f27faf435f81e076bfc2ef0795" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:00:58.590381 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647f3f27faf435f81e076bfc2ef0795" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:00:58.590603 2538 raft_consensus.cc:2454] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:58.590662 5161 raft_consensus.cc:2454] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:00:58.591076 4967 leader_election.cc:302] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:58.591389 8387 raft_consensus.cc:2781] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:58.591502 8387 raft_consensus.cc:482] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:58.591567 8387 raft_consensus.cc:3037] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.593693 8387 raft_consensus.cc:504] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.594331 8387 leader_election.cc:288] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:00:58.594810 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647f3f27faf435f81e076bfc2ef0795" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:00:58.594885 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647f3f27faf435f81e076bfc2ef0795" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:00:58.595052 2538 raft_consensus.cc:3037] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.595134 5161 raft_consensus.cc:3037] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.597848 2538 raft_consensus.cc:2454] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:58.597853 5161 raft_consensus.cc:2454] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:00:58.598296 4967 leader_election.cc:302] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:00:58.598661 8387 raft_consensus.cc:2781] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:58.598802 8387 raft_consensus.cc:686] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:00:58.599098 8387 consensus_queue.cc:237] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:00:58.604266 2000 catalog_manager.cc:5484] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:58.649741 8161 raft_consensus.cc:482] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:58.649928 8161 raft_consensus.cc:504] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.650588 8161 leader_election.cc:288] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:58.651353 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e081778a0f67486a904f3bed7dfe8644" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:58.651608 5296 raft_consensus.cc:2454] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:58.652060 2473 leader_election.cc:302] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:58.652088 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e081778a0f67486a904f3bed7dfe8644" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:58.652318 5032 raft_consensus.cc:2454] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:58.652412 8161 raft_consensus.cc:2781] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:58.652487 8161 raft_consensus.cc:482] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:58.652558 8161 raft_consensus.cc:3037] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.656726 8161 raft_consensus.cc:504] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.658159 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e081778a0f67486a904f3bed7dfe8644" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:58.658381 5296 raft_consensus.cc:3037] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.658376 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e081778a0f67486a904f3bed7dfe8644" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:58.658594 5032 raft_consensus.cc:3037] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:58.661502 5032 raft_consensus.cc:2454] T e081778a0f67486a904f3bed7dfe8644 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:58.662055 2473 leader_election.cc:302] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:00:58.662398 8413 raft_consensus.cc:2781] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:58.662477 8161 leader_election.cc:288] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:58.662588 8413 raft_consensus.cc:686] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:58.662946 8413 consensus_queue.cc:237] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:58.665402 5296 raft_consensus.cc:2454] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:58.669957 2000 catalog_manager.cc:5484] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:58.673259 2003 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:58.684877 8408 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:58.685370 8409 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:58.698853 8410 tablet.cc:1722] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:58.704015 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=d6ac5c5b725641afa22a7b0464219091] alter complete (version 3)
I20240501 14:00:58.808441 2000 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:58.808670 2000 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:58.815373 2000 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 580da5dc03db41d89ff3b7a4a97927b7
I20240501 14:00:58.823094 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet 580da5dc03db41d89ff3b7a4a97927b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:00:58.823614 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 580da5dc03db41d89ff3b7a4a97927b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:58.823891 8418 tablet_replica.cc:330] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:00:58.824003 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 580da5dc03db41d89ff3b7a4a97927b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:58 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:00:58.824177 8418 raft_consensus.cc:2229] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:58.824575 8419 tablet_replica.cc:330] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:58.824616 8418 raft_consensus.cc:2258] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.824817 8419 raft_consensus.cc:2229] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:58.825027 8419 raft_consensus.cc:2258] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.825910 8418 ts_tablet_manager.cc:1875] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.828315 8419 ts_tablet_manager.cc:1875] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.831622 8420 tablet_replica.cc:330] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:00:58.831848 8420 raft_consensus.cc:2229] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:58.832093 8420 raft_consensus.cc:2258] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:58.832752 8418 ts_tablet_manager.cc:1888] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.832911 8418 log.cc:1196] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/580da5dc03db41d89ff3b7a4a97927b7
I20240501 14:00:58.833283 8420 ts_tablet_manager.cc:1875] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:58.833357 8418 ts_tablet_manager.cc:1909] T 580da5dc03db41d89ff3b7a4a97927b7 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:00:58.834570 8419 ts_tablet_manager.cc:1888] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.834560 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 580da5dc03db41d89ff3b7a4a97927b7 (table rebalancer_dynamic_table_new_name [id=d6ac5c5b725641afa22a7b0464219091]) successfully deleted
I20240501 14:00:58.834733 8419 log.cc:1196] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/580da5dc03db41d89ff3b7a4a97927b7
I20240501 14:00:58.835503 8419 ts_tablet_manager.cc:1909] T 580da5dc03db41d89ff3b7a4a97927b7 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:58.836675 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 580da5dc03db41d89ff3b7a4a97927b7 (table rebalancer_dynamic_table_new_name [id=d6ac5c5b725641afa22a7b0464219091]) successfully deleted
I20240501 14:00:58.837980 8420 ts_tablet_manager.cc:1888] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:58.838101 8420 log.cc:1196] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/580da5dc03db41d89ff3b7a4a97927b7
I20240501 14:00:58.838544 8420 ts_tablet_manager.cc:1909] T 580da5dc03db41d89ff3b7a4a97927b7 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:00:58.839784 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 580da5dc03db41d89ff3b7a4a97927b7 (table rebalancer_dynamic_table_new_name [id=d6ac5c5b725641afa22a7b0464219091]) successfully deleted
I20240501 14:00:58.925091 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:58.932621 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]), partition=
I20240501 14:00:58.933005 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]), partition=
I20240501 14:00:58.932987 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]), partition=
I20240501 14:00:58.933041 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331ac735a09b48b3a8f9b7abcd2d0a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.933360 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331ac735a09b48b3a8f9b7abcd2d0a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.933372 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331ac735a09b48b3a8f9b7abcd2d0a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:58.937602 8421 tablet_bootstrap.cc:492] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:00:58.939714 8421 tablet_bootstrap.cc:654] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.944730 8421 tablet_bootstrap.cc:492] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:00:58.945780 8423 tablet_bootstrap.cc:492] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:58.946199 8421 ts_tablet_manager.cc:1367] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 14:00:58.946985 8422 tablet_bootstrap.cc:492] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:58.947556 8421 raft_consensus.cc:348] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:58.947785 8423 tablet_bootstrap.cc:654] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.947791 8421 raft_consensus.cc:374] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.947880 8421 raft_consensus.cc:729] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.948168 8421 consensus_queue.cc:260] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:58.948734 8422 tablet_bootstrap.cc:654] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:58.948889 8421 ts_tablet_manager.cc:1398] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:58.951779 8422 tablet_bootstrap.cc:492] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:58.951985 8422 ts_tablet_manager.cc:1367] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:00:58.952802 8422 raft_consensus.cc:348] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:58.952991 8422 raft_consensus.cc:374] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.953145 8422 raft_consensus.cc:729] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.953425 8422 consensus_queue.cc:260] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:58.954305 8422 ts_tablet_manager.cc:1398] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:58.955914 8423 tablet_bootstrap.cc:492] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:58.956079 8423 ts_tablet_manager.cc:1367] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240501 14:00:58.956871 8423 raft_consensus.cc:348] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:58.957094 8423 raft_consensus.cc:374] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:58.957198 8423 raft_consensus.cc:729] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:58.957439 8423 consensus_queue.cc:260] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:58.959558 8423 ts_tablet_manager.cc:1398] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:00:59.017714 8307 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.023299 8307 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.081666 8307 consensus_queue.cc:1035] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.086557 8413 consensus_queue.cc:1035] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.100353 8385 raft_consensus.cc:482] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:59.100564 8385 raft_consensus.cc:504] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:59.101322 8385 leader_election.cc:288] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:59.101851 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331ac735a09b48b3a8f9b7abcd2d0a54" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:59.101855 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331ac735a09b48b3a8f9b7abcd2d0a54" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:00:59.102073 2408 raft_consensus.cc:2454] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:59.102088 5292 raft_consensus.cc:2454] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:00:59.102485 2080 leader_election.cc:302] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:59.102809 8385 raft_consensus.cc:2781] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:59.102911 8385 raft_consensus.cc:482] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:59.103003 8385 raft_consensus.cc:3037] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.105459 8385 raft_consensus.cc:504] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:59.106067 8385 leader_election.cc:288] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:59.106555 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331ac735a09b48b3a8f9b7abcd2d0a54" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:59.106552 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331ac735a09b48b3a8f9b7abcd2d0a54" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:00:59.106746 2408 raft_consensus.cc:3037] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.106765 5292 raft_consensus.cc:3037] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.108772 2408 raft_consensus.cc:2454] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:59.108784 5292 raft_consensus.cc:2454] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:00:59.109166 2082 leader_election.cc:302] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:59.109462 8385 raft_consensus.cc:2781] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:59.109724 8385 raft_consensus.cc:686] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:00:59.109989 8385 consensus_queue.cc:237] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:59.114591 2005 catalog_manager.cc:5484] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:59.204146 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:59.204392 8387 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.216794 2517 tablet_service.cc:1467] Processing CreateTablet for tablet e4a0a92e0d48493fac254f1b249a63cf (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.216761 2257 tablet_service.cc:1467] Processing CreateTablet for tablet e4a0a92e0d48493fac254f1b249a63cf (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.217217 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a0a92e0d48493fac254f1b249a63cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.217233 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a0a92e0d48493fac254f1b249a63cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.217290 8286 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.218813 5013 tablet_service.cc:1467] Processing CreateTablet for tablet afc49574262e4819b6c86025c43c772a (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.219300 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc49574262e4819b6c86025c43c772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.219388 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 14e43f9e141140be88a7dc1fa8963c49 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.219758 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e43f9e141140be88a7dc1fa8963c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.223331 5014 tablet_service.cc:1467] Processing CreateTablet for tablet 14e43f9e141140be88a7dc1fa8963c49 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.223776 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e43f9e141140be88a7dc1fa8963c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.224129 2125 tablet_service.cc:1467] Processing CreateTablet for tablet e4a0a92e0d48493fac254f1b249a63cf (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.224525 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a0a92e0d48493fac254f1b249a63cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.227339 4879 tablet_service.cc:1467] Processing CreateTablet for tablet afc49574262e4819b6c86025c43c772a (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.227756 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc49574262e4819b6c86025c43c772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.231827 2127 tablet_service.cc:1467] Processing CreateTablet for tablet afc49574262e4819b6c86025c43c772a (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.232297 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc49574262e4819b6c86025c43c772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.233395 8422 tablet_bootstrap.cc:492] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:59.235029 8422 tablet_bootstrap.cc:654] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.236984 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 14e43f9e141140be88a7dc1fa8963c49 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.236934 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:00:59.237399 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e43f9e141140be88a7dc1fa8963c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.238792 8423 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:59.240326 8423 tablet_bootstrap.cc:654] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.241053 8437 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:59.242619 8437 tablet_bootstrap.cc:654] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.243832 8442 tablet_bootstrap.cc:492] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:59.245991 8442 tablet_bootstrap.cc:654] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.253567 8444 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:59.256565 8444 tablet_bootstrap.cc:654] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.268785 8423 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:59.270164 8444 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:59.271623 8423 ts_tablet_manager.cc:1367] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.001s
I20240501 14:00:59.272989 8423 raft_consensus.cc:348] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.276103 8423 raft_consensus.cc:374] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.276208 8423 raft_consensus.cc:729] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.276921 8444 ts_tablet_manager.cc:1367] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.005s
I20240501 14:00:59.277873 8444 raft_consensus.cc:348] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.278174 8444 raft_consensus.cc:374] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.278744 8444 raft_consensus.cc:729] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.279675 8423 consensus_queue.cc:260] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.281545 8444 consensus_queue.cc:260] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.280251 8423 ts_tablet_manager.cc:1398] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:00:59.282263 8423 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:59.285075 8444 ts_tablet_manager.cc:1398] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:00:59.285248 8437 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:59.285400 8437 ts_tablet_manager.cc:1367] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.006s
I20240501 14:00:59.285728 8422 tablet_bootstrap.cc:492] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:59.285874 8422 ts_tablet_manager.cc:1367] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.005s
I20240501 14:00:59.286048 8437 raft_consensus.cc:348] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.286370 8437 raft_consensus.cc:374] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.286461 8437 raft_consensus.cc:729] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.286803 8437 consensus_queue.cc:260] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.289201 8422 raft_consensus.cc:348] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.289753 8423 tablet_bootstrap.cc:654] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.289772 8422 raft_consensus.cc:374] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.289808 8437 ts_tablet_manager.cc:1398] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:59.290011 8422 raft_consensus.cc:729] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.290050 8437 tablet_bootstrap.cc:492] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:59.290310 8422 consensus_queue.cc:260] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.291931 8422 ts_tablet_manager.cc:1398] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:59.292084 8437 tablet_bootstrap.cc:654] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.294631 5292 raft_consensus.cc:1261] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:59.296360 8388 consensus_queue.cc:1035] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.298408 8423 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:59.298650 8423 ts_tablet_manager.cc:1367] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20240501 14:00:59.299520 8423 raft_consensus.cc:348] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.300971 8423 raft_consensus.cc:374] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.301131 8423 raft_consensus.cc:729] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.301437 8423 consensus_queue.cc:260] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.301862 8437 tablet_bootstrap.cc:492] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:59.302057 8437 ts_tablet_manager.cc:1367] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240501 14:00:59.302157 8423 ts_tablet_manager.cc:1398] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:00:59.303360 8437 raft_consensus.cc:348] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.304224 8437 raft_consensus.cc:374] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.304340 8437 raft_consensus.cc:729] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.304623 8437 consensus_queue.cc:260] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.306558 8437 ts_tablet_manager.cc:1398] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:59.317132 8442 tablet_bootstrap.cc:492] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:59.318166 8442 ts_tablet_manager.cc:1367] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.000s
I20240501 14:00:59.319273 8442 raft_consensus.cc:348] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.321421 8442 raft_consensus.cc:374] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.321733 8442 raft_consensus.cc:729] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.322341 8442 consensus_queue.cc:260] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.324646 8442 ts_tablet_manager.cc:1398] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:59.325511 8442 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:00:59.325685 8438 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:00:59.326964 8442 tablet_bootstrap.cc:654] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.327354 8438 tablet_bootstrap.cc:654] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.329506 8442 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:00:59.329891 8442 ts_tablet_manager.cc:1367] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:59.330860 8442 raft_consensus.cc:348] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.331231 8442 raft_consensus.cc:374] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.331359 8442 raft_consensus.cc:729] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.331764 8442 consensus_queue.cc:260] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.332613 8442 ts_tablet_manager.cc:1398] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:59.355983 2408 raft_consensus.cc:1261] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:59.357959 8388 consensus_queue.cc:1035] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.365420 2002 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]). Expected version 1 got 0
I20240501 14:00:59.370075 8438 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:00:59.370280 8438 ts_tablet_manager.cc:1367] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.016s
I20240501 14:00:59.371081 8438 raft_consensus.cc:348] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.371318 8438 raft_consensus.cc:374] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.371413 8438 raft_consensus.cc:729] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.371662 8438 consensus_queue.cc:260] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.379707 8455 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:59.380359 8455 tablet.cc:1620] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:00:59.377651 8438 ts_tablet_manager.cc:1398] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:00:59.382628 2002 catalog_manager.cc:5576] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309) does not have the latest schema for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]). Expected version 1 got 0
I20240501 14:00:59.385025 8456 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:59.385704 8456 tablet.cc:1620] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:59.389662 2005 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]). Expected version 1 got 0
I20240501 14:00:59.390372 8161 raft_consensus.cc:482] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:59.390578 8161 raft_consensus.cc:504] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.391250 8161 leader_election.cc:288] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:59.391749 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a0a92e0d48493fac254f1b249a63cf" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:00:59.391984 2274 raft_consensus.cc:2454] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:59.392791 2471 leader_election.cc:302] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:59.393086 8161 raft_consensus.cc:2781] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:59.393179 8161 raft_consensus.cc:482] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:59.393280 8161 raft_consensus.cc:3037] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.395128 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a0a92e0d48493fac254f1b249a63cf" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:59.395372 2143 raft_consensus.cc:2454] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:00:59.395857 8161 raft_consensus.cc:504] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.396569 8161 leader_election.cc:288] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:00:59.397437 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a0a92e0d48493fac254f1b249a63cf" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:00:59.397626 2274 raft_consensus.cc:3037] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.399674 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a0a92e0d48493fac254f1b249a63cf" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:59.399874 2143 raft_consensus.cc:3037] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.402427 8364 raft_consensus.cc:482] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:59.402652 8364 raft_consensus.cc:504] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.403348 8364 leader_election.cc:288] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:59.403385 2143 raft_consensus.cc:2454] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:59.404515 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc49574262e4819b6c86025c43c772a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:59.404803 5031 raft_consensus.cc:2454] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:59.405535 4836 leader_election.cc:302] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:59.406004 7862 raft_consensus.cc:482] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:59.406154 7862 raft_consensus.cc:504] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.406333 8364 raft_consensus.cc:2781] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:59.406450 8364 raft_consensus.cc:482] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:59.403818 2471 leader_election.cc:302] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:00:59.407048 7862 leader_election.cc:288] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:59.407477 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e43f9e141140be88a7dc1fa8963c49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:00:59.407562 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e43f9e141140be88a7dc1fa8963c49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:00:59.407716 5031 raft_consensus.cc:2454] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:59.407774 2406 raft_consensus.cc:2454] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:00:59.408185 2213 leader_election.cc:302] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:59.408427 7862 raft_consensus.cc:2781] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:59.408483 7862 raft_consensus.cc:482] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:59.408531 7862 raft_consensus.cc:3037] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.408818 8307 raft_consensus.cc:2781] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:59.408869 8364 raft_consensus.cc:3037] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.408964 8307 raft_consensus.cc:686] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:00:59.409237 8307 consensus_queue.cc:237] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:00:59.411540 8364 raft_consensus.cc:504] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.411825 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc49574262e4819b6c86025c43c772a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:00:59.412130 2143 raft_consensus.cc:2454] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:00:59.415360 1997 catalog_manager.cc:5484] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:59.417565 8364 leader_election.cc:288] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:00:59.418802 8460 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:00:59.418954 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574] alter complete (version 1)
I20240501 14:00:59.419548 8460 tablet.cc:1620] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:59.421602 2005 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]). Expected version 1 got 0
I20240501 14:00:59.422142 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc49574262e4819b6c86025c43c772a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:59.422421 5031 raft_consensus.cc:3037] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.425189 5031 raft_consensus.cc:2454] T afc49574262e4819b6c86025c43c772a P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:59.425808 4836 leader_election.cc:302] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:00:59.430581 2274 raft_consensus.cc:2454] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:00:59.431103 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc49574262e4819b6c86025c43c772a" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:00:59.431365 8364 raft_consensus.cc:2781] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:59.431528 8364 raft_consensus.cc:686] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:00:59.431799 8364 consensus_queue.cc:237] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.431947 7862 raft_consensus.cc:504] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.432472 2143 raft_consensus.cc:3037] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.433122 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e43f9e141140be88a7dc1fa8963c49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:00:59.433383 5032 raft_consensus.cc:3037] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.434638 2143 raft_consensus.cc:2454] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:00:59.436007 5032 raft_consensus.cc:2454] T 14e43f9e141140be88a7dc1fa8963c49 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:59.436375 7862 leader_election.cc:288] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:00:59.436589 2213 leader_election.cc:302] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:00:59.436802 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e43f9e141140be88a7dc1fa8963c49" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:00:59.437052 2408 raft_consensus.cc:3037] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:59.438285 8366 raft_consensus.cc:2781] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:59.439998 2408 raft_consensus.cc:2454] T 14e43f9e141140be88a7dc1fa8963c49 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:00:59.441038 8366 raft_consensus.cc:686] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:00:59.441318 8366 consensus_queue.cc:237] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:00:59.443560 2002 catalog_manager.cc:5484] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:59.445416 2005 catalog_manager.cc:5484] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:59.523660 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:00:59.533933 8455 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:59.534447 8455 tablet.cc:1620] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:00:59.534636 8456 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:59.535074 8456 tablet.cc:1620] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:00:59.542551 8460 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:00:59.543125 8460 tablet.cc:1620] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:00:59.543474 2002 catalog_manager.cc:5576] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309) does not have the latest schema for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574]). Expected version 2 got 1
I20240501 14:00:59.544881 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=2998bbc052b443cfbfc8156dd2ddc574] alter complete (version 2)
I20240501 14:00:59.647574 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:00:59.656497 8455 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:59.657869 8460 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:59.662732 8456 tablet.cc:1722] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:00:59.674052 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=2998bbc052b443cfbfc8156dd2ddc574] alter complete (version 3)
I20240501 14:00:59.779780 2000 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:59.780150 2000 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:00:59.783843 2000 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 331ac735a09b48b3a8f9b7abcd2d0a54
I20240501 14:00:59.784718 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:59 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:00:59.784767 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:59 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:00:59.785157 2127 tablet_service.cc:1514] Processing DeleteTablet for tablet 331ac735a09b48b3a8f9b7abcd2d0a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:00:59 UTC) from {username='slave'} at 127.0.0.1:48706
I20240501 14:00:59.785527 8466 tablet_replica.cc:330] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:00:59.785794 8466 raft_consensus.cc:2229] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:59.786026 8466 raft_consensus.cc:2258] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.787021 8466 ts_tablet_manager.cc:1875] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:59.795285 8467 tablet_replica.cc:330] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:00:59.795367 8466 ts_tablet_manager.cc:1888] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:59.795521 8466 log.cc:1196] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/331ac735a09b48b3a8f9b7abcd2d0a54
I20240501 14:00:59.795668 8467 raft_consensus.cc:2229] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:59.795946 8466 ts_tablet_manager.cc:1909] T 331ac735a09b48b3a8f9b7abcd2d0a54 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:00:59.796054 8467 raft_consensus.cc:2258] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.796977 8467 ts_tablet_manager.cc:1875] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:59.797204 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table_new_name [id=2998bbc052b443cfbfc8156dd2ddc574]) successfully deleted
I20240501 14:00:59.797658 8465 tablet_replica.cc:330] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:00:59.797989 8465 raft_consensus.cc:2229] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:59.798246 8465 raft_consensus.cc:2258] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.799165 8465 ts_tablet_manager.cc:1875] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:00:59.802003 8467 ts_tablet_manager.cc:1888] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:59.802111 8467 log.cc:1196] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/331ac735a09b48b3a8f9b7abcd2d0a54
I20240501 14:00:59.802448 8467 ts_tablet_manager.cc:1909] T 331ac735a09b48b3a8f9b7abcd2d0a54 P d1a414d381f44829a2510cb76d481887: Deleting consensus metadata
I20240501 14:00:59.803517 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table_new_name [id=2998bbc052b443cfbfc8156dd2ddc574]) successfully deleted
I20240501 14:00:59.804059 8465 ts_tablet_manager.cc:1888] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:00:59.804179 8465 log.cc:1196] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/331ac735a09b48b3a8f9b7abcd2d0a54
I20240501 14:00:59.804549 8465 ts_tablet_manager.cc:1909] T 331ac735a09b48b3a8f9b7abcd2d0a54 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:00:59.805523 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 331ac735a09b48b3a8f9b7abcd2d0a54 (table rebalancer_dynamic_table_new_name [id=2998bbc052b443cfbfc8156dd2ddc574]) successfully deleted
I20240501 14:00:59.847811 8366 consensus_queue.cc:1035] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.854599 8462 consensus_queue.cc:1035] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.856834 8413 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.874293 8161 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.886703 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:00:59.896418 2388 tablet_service.cc:1467] Processing CreateTablet for tablet ccae5cec7fdf454d81282b311e948d59 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5]), partition=
I20240501 14:00:59.896834 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccae5cec7fdf454d81282b311e948d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.898955 2257 tablet_service.cc:1467] Processing CreateTablet for tablet ccae5cec7fdf454d81282b311e948d59 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5]), partition=
I20240501 14:00:59.899350 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccae5cec7fdf454d81282b311e948d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.899588 8364 consensus_queue.cc:1035] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.901451 4879 tablet_service.cc:1467] Processing CreateTablet for tablet ccae5cec7fdf454d81282b311e948d59 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5]), partition=
I20240501 14:00:59.901803 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccae5cec7fdf454d81282b311e948d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.909691 8478 tablet_bootstrap.cc:492] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:00:59.911500 8478 tablet_bootstrap.cc:654] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.912297 8150 consensus_queue.cc:1035] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:59.935541 8479 tablet_bootstrap.cc:492] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:00:59.938578 8476 tablet_bootstrap.cc:492] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:00:59.942927 8478 tablet_bootstrap.cc:492] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:00:59.943173 8478 ts_tablet_manager.cc:1367] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.007s
I20240501 14:00:59.943303 8476 tablet_bootstrap.cc:654] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.943979 8478 raft_consensus.cc:348] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.944240 8478 raft_consensus.cc:374] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.944347 8478 raft_consensus.cc:729] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.944433 8479 tablet_bootstrap.cc:654] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.944629 8478 consensus_queue.cc:260] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.946601 8478 ts_tablet_manager.cc:1398] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:59.955343 8479 tablet_bootstrap.cc:492] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:00:59.955617 8479 ts_tablet_manager.cc:1367] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240501 14:00:59.955925 8476 tablet_bootstrap.cc:492] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:00:59.956112 8476 ts_tablet_manager.cc:1367] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240501 14:00:59.956857 8476 raft_consensus.cc:348] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.957137 8476 raft_consensus.cc:374] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.957242 8476 raft_consensus.cc:729] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.957553 8476 consensus_queue.cc:260] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.958154 8479 raft_consensus.cc:348] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.958593 8479 raft_consensus.cc:374] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.958742 8479 raft_consensus.cc:729] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.958726 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:00:59.959471 8476 ts_tablet_manager.cc:1398] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:59.961134 8479 consensus_queue.cc:260] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:00:59.971877 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff679e55a5446099efbdbcd1109f245 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.972429 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff679e55a5446099efbdbcd1109f245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.973299 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff679e55a5446099efbdbcd1109f245 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.973299 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 34b137c6ac1947cd80aee3c6a6f9d138 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.973706 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b137c6ac1947cd80aee3c6a6f9d138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.973724 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff679e55a5446099efbdbcd1109f245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.974542 8479 ts_tablet_manager.cc:1398] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240501 14:00:59.974879 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff679e55a5446099efbdbcd1109f245 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.974905 2517 tablet_service.cc:1467] Processing CreateTablet for tablet e3fe9fe1c13743478e09f5eeeb528b57 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.975208 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff679e55a5446099efbdbcd1109f245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.975299 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fe9fe1c13743478e09f5eeeb528b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.978905 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 34b137c6ac1947cd80aee3c6a6f9d138 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.979329 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b137c6ac1947cd80aee3c6a6f9d138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.979310 2125 tablet_service.cc:1467] Processing CreateTablet for tablet e3fe9fe1c13743478e09f5eeeb528b57 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.979679 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fe9fe1c13743478e09f5eeeb528b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.983446 2388 tablet_service.cc:1467] Processing CreateTablet for tablet e3fe9fe1c13743478e09f5eeeb528b57 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.983879 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3fe9fe1c13743478e09f5eeeb528b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.986882 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 34b137c6ac1947cd80aee3c6a6f9d138 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:59.987054 8485 tablet_bootstrap.cc:492] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:59.987329 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b137c6ac1947cd80aee3c6a6f9d138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.988579 8485 tablet_bootstrap.cc:654] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:59.991917 8485 tablet_bootstrap.cc:492] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:00:59.992159 8485 ts_tablet_manager.cc:1367] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 14:00:59.992874 8485 raft_consensus.cc:348] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:59.993141 8485 raft_consensus.cc:374] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:59.993225 8485 raft_consensus.cc:729] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:00:59.993477 8485 consensus_queue.cc:260] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:00:59.995558 8485 ts_tablet_manager.cc:1398] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:00:59.995889 8485 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:00:59.997470 8485 tablet_bootstrap.cc:654] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.002254 8479 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:00.002359 8488 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:00.003989 8479 tablet_bootstrap.cc:654] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.004170 8488 tablet_bootstrap.cc:654] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.004791 8478 tablet_bootstrap.cc:492] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:00.006373 8478 tablet_bootstrap.cc:654] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.008560 8476 tablet_bootstrap.cc:492] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:00.010569 8476 tablet_bootstrap.cc:654] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.013438 8485 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:01:00.013797 8485 ts_tablet_manager.cc:1367] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20240501 14:01:00.013919 8478 tablet_bootstrap.cc:492] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:00.014088 8478 ts_tablet_manager.cc:1367] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240501 14:01:00.014918 8478 raft_consensus.cc:348] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.014991 8485 raft_consensus.cc:348] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.015154 8478 raft_consensus.cc:374] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.015177 8485 raft_consensus.cc:374] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.015177 8486 tablet_bootstrap.cc:492] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:00.015218 8478 raft_consensus.cc:729] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.015226 8485 raft_consensus.cc:729] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.015465 8485 consensus_queue.cc:260] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.015445 8478 consensus_queue.cc:260] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.016489 8478 ts_tablet_manager.cc:1398] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:00.017280 8478 tablet_bootstrap.cc:492] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:00.018191 8486 tablet_bootstrap.cc:654] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.018826 8485 ts_tablet_manager.cc:1398] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:01:00.019048 8478 tablet_bootstrap.cc:654] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.021984 8476 tablet_bootstrap.cc:492] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:01:00.022153 8476 ts_tablet_manager.cc:1367] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240501 14:01:00.023317 8476 raft_consensus.cc:348] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.023605 8476 raft_consensus.cc:374] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.023701 8476 raft_consensus.cc:729] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.023979 8476 consensus_queue.cc:260] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.026165 8476 ts_tablet_manager.cc:1398] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:00.029265 8478 tablet_bootstrap.cc:492] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:00.029469 8478 ts_tablet_manager.cc:1367] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 14:01:00.030258 8478 raft_consensus.cc:348] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.030542 8478 raft_consensus.cc:374] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.030618 8478 raft_consensus.cc:729] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.030848 8478 consensus_queue.cc:260] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.031443 8478 ts_tablet_manager.cc:1398] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:00.037647 8486 tablet_bootstrap.cc:492] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:01:00.037936 8486 ts_tablet_manager.cc:1367] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.008s
I20240501 14:01:00.038383 8479 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:01:00.038596 8479 ts_tablet_manager.cc:1367] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.006s
I20240501 14:01:00.038981 8486 raft_consensus.cc:348] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.039285 8486 raft_consensus.cc:374] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.039425 8486 raft_consensus.cc:729] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.039458 8479 raft_consensus.cc:348] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.039696 8479 raft_consensus.cc:374] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.039714 8488 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:00.039773 8479 raft_consensus.cc:729] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.039871 8488 ts_tablet_manager.cc:1367] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.007s
I20240501 14:01:00.039894 8486 consensus_queue.cc:260] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.040108 8479 consensus_queue.cc:260] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.041364 8479 ts_tablet_manager.cc:1398] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:00.041993 8488 raft_consensus.cc:348] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.042210 8488 raft_consensus.cc:374] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.042308 8488 raft_consensus.cc:729] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.042881 8488 consensus_queue.cc:260] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.044893 8486 ts_tablet_manager.cc:1398] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:01:00.045886 8488 ts_tablet_manager.cc:1398] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240501 14:01:00.047403 8488 tablet_bootstrap.cc:492] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:00.049248 8488 tablet_bootstrap.cc:654] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.053090 8488 tablet_bootstrap.cc:492] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:00.053367 8488 ts_tablet_manager.cc:1367] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240501 14:01:00.054314 8488 raft_consensus.cc:348] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.054597 8488 raft_consensus.cc:374] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.054783 8488 raft_consensus.cc:729] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.055095 8488 consensus_queue.cc:260] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.055760 8488 ts_tablet_manager.cc:1398] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:00.074066 8385 raft_consensus.cc:482] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.074230 8385 raft_consensus.cc:504] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.074864 8385 leader_election.cc:288] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:00.075385 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe9fe1c13743478e09f5eeeb528b57" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:00.075426 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe9fe1c13743478e09f5eeeb528b57" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:00.075628 2537 raft_consensus.cc:2454] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:01:00.075651 2408 raft_consensus.cc:2454] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:01:00.075963 2080 leader_election.cc:302] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:00.076186 8385 raft_consensus.cc:2781] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.076256 8385 raft_consensus.cc:482] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.076328 8385 raft_consensus.cc:3037] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.078246 8385 raft_consensus.cc:504] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.078770 8385 leader_election.cc:288] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:00.079232 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe9fe1c13743478e09f5eeeb528b57" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:00.079373 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fe9fe1c13743478e09f5eeeb528b57" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:00.079488 2537 raft_consensus.cc:3037] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.079555 2408 raft_consensus.cc:3037] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.082242 2408 raft_consensus.cc:2454] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:00.082253 2537 raft_consensus.cc:2454] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:00.082839 2080 leader_election.cc:302] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:00.083081 8385 raft_consensus.cc:2781] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.083233 8385 raft_consensus.cc:686] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:00.083494 8385 consensus_queue.cc:237] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.087591 8480 raft_consensus.cc:482] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.087766 8480 raft_consensus.cc:504] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.088567 8480 leader_election.cc:288] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:00.089290 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccae5cec7fdf454d81282b311e948d59" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:00.089514 2274 raft_consensus.cc:2454] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:01:00.089905 4834 leader_election.cc:302] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:00.090242 8480 raft_consensus.cc:2781] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.090363 8480 raft_consensus.cc:482] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.090312 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccae5cec7fdf454d81282b311e948d59" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:00.090443 8480 raft_consensus.cc:3037] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.090566 2408 raft_consensus.cc:2454] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:01:00.091495 2000 catalog_manager.cc:5484] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:00.093298 8480 raft_consensus.cc:504] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.093955 8480 leader_election.cc:288] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:00.094759 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccae5cec7fdf454d81282b311e948d59" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:00.094935 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccae5cec7fdf454d81282b311e948d59" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:01:00.095110 2274 raft_consensus.cc:3037] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.095130 2408 raft_consensus.cc:3037] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.097847 2408 raft_consensus.cc:2454] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:00.097846 2274 raft_consensus.cc:2454] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:00.098354 4834 leader_election.cc:302] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:00.099740 8480 raft_consensus.cc:2781] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.100095 8480 raft_consensus.cc:686] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:00.100483 8480 consensus_queue.cc:237] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.104960 2000 catalog_manager.cc:5484] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:00.172859 7872 raft_consensus.cc:482] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.173064 7872 raft_consensus.cc:504] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.173815 7872 leader_election.cc:288] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:00.174202 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b137c6ac1947cd80aee3c6a6f9d138" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:00.174274 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b137c6ac1947cd80aee3c6a6f9d138" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:00.174407 2408 raft_consensus.cc:2454] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:00.174537 2274 raft_consensus.cc:2454] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:00.174856 5097 leader_election.cc:302] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:00.175093 7872 raft_consensus.cc:2781] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.175204 7872 raft_consensus.cc:482] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.175282 7872 raft_consensus.cc:3037] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.177203 7872 raft_consensus.cc:504] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.177747 7872 leader_election.cc:288] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:00.178287 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b137c6ac1947cd80aee3c6a6f9d138" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:00.178290 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b137c6ac1947cd80aee3c6a6f9d138" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:01:00.178490 2274 raft_consensus.cc:3037] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.178530 2408 raft_consensus.cc:3037] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.180841 2274 raft_consensus.cc:2454] T 34b137c6ac1947cd80aee3c6a6f9d138 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:00.180903 2408 raft_consensus.cc:2454] T 34b137c6ac1947cd80aee3c6a6f9d138 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:00.181435 5097 leader_election.cc:302] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:00.181797 7872 raft_consensus.cc:2781] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.181988 7872 raft_consensus.cc:686] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:01:00.182322 7872 consensus_queue.cc:237] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.186610 2002 catalog_manager.cc:5484] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.234278 7872 raft_consensus.cc:482] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.234709 7872 raft_consensus.cc:504] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.235471 7872 leader_election.cc:288] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:00.235867 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff679e55a5446099efbdbcd1109f245" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:01:00.236045 2145 raft_consensus.cc:2454] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:00.236002 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff679e55a5446099efbdbcd1109f245" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:01:00.236218 4899 raft_consensus.cc:2454] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:00.236390 5097 leader_election.cc:302] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:00.236878 7872 raft_consensus.cc:2781] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.236961 7872 raft_consensus.cc:482] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.237080 7872 raft_consensus.cc:3037] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.239207 7872 raft_consensus.cc:504] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.240090 7872 leader_election.cc:288] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:00.240053 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:00.240994 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff679e55a5446099efbdbcd1109f245" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:01:00.241216 4899 raft_consensus.cc:3037] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.241230 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff679e55a5446099efbdbcd1109f245" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:01:00.241401 2143 raft_consensus.cc:3037] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.243800 2143 raft_consensus.cc:2454] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:00.243831 4899 raft_consensus.cc:2454] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:00.244890 5098 leader_election.cc:302] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:00.247774 2277 raft_consensus.cc:1261] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:00.247993 7872 raft_consensus.cc:2781] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.248165 7872 raft_consensus.cc:686] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:01:00.248445 7872 consensus_queue.cc:237] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:00.248626 8150 consensus_queue.cc:1035] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.249747 2408 raft_consensus.cc:1261] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:00.250463 8480 consensus_queue.cc:1035] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 14:01:00.256007 8505 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:00.256759 2005 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet ccae5cec7fdf454d81282b311e948d59 (table rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5]). Expected version 1 got 0
I20240501 14:01:00.256883 8505 tablet.cc:1620] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:01:00.261916 2000 catalog_manager.cc:5484] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.264487 8507 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:00.265373 8507 tablet.cc:1620] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:01:00.266419 2005 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet ccae5cec7fdf454d81282b311e948d59 (table rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5]). Expected version 1 got 0
I20240501 14:01:00.272107 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5] alter complete (version 1)
I20240501 14:01:00.275327 2005 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet ccae5cec7fdf454d81282b311e948d59 (table rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5]). Expected version 1 got 0
I20240501 14:01:00.277791 8506 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:00.278482 8506 tablet.cc:1620] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:01:00.379248 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:00.391530 8505 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:00.392076 8505 tablet.cc:1620] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: MemRowSet was empty: no flush needed.
I20240501 14:01:00.397274 8506 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:00.397717 8506 tablet.cc:1620] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:01:00.402647 8507 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:00.403379 8507 tablet.cc:1620] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:01:00.407953 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=85b95af3ee2c400b9775242e3dfd7dc5] alter complete (version 2)
I20240501 14:01:00.513746 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:00.521945 8505 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:00.523018 8506 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:00.529958 8507 tablet.cc:1722] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:00.533624 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=85b95af3ee2c400b9775242e3dfd7dc5] alter complete (version 3)
I20240501 14:01:00.568295 7872 consensus_queue.cc:1035] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.583268 8514 consensus_queue.cc:1035] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.593595 8385 consensus_queue.cc:1035] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.608861 8385 consensus_queue.cc:1035] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.639057 2002 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:00.639320 2002 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:00.644999 2002 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet ccae5cec7fdf454d81282b311e948d59
I20240501 14:01:00.645866 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet ccae5cec7fdf454d81282b311e948d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:00 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:01:00.646245 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet ccae5cec7fdf454d81282b311e948d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:00 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:01:00.646395 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet ccae5cec7fdf454d81282b311e948d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:00 UTC) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:00.659324 8520 tablet_replica.cc:330] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:01:00.659690 8520 raft_consensus.cc:2229] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:00.659967 8520 raft_consensus.cc:2258] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:00.661088 8520 ts_tablet_manager.cc:1875] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:00.661656 8522 tablet_replica.cc:330] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:00.661870 8521 tablet_replica.cc:330] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:01:00.661901 8522 raft_consensus.cc:2229] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:00.662171 8521 raft_consensus.cc:2229] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:00.662436 8521 raft_consensus.cc:2258] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:00.662438 8522 raft_consensus.cc:2258] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:00.663228 8522 ts_tablet_manager.cc:1875] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:00.663662 8521 ts_tablet_manager.cc:1875] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:00.669929 8521 ts_tablet_manager.cc:1888] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:00.670104 8521 log.cc:1196] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/ccae5cec7fdf454d81282b311e948d59
I20240501 14:01:00.670492 8521 ts_tablet_manager.cc:1909] T ccae5cec7fdf454d81282b311e948d59 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:01:00.670907 8522 ts_tablet_manager.cc:1888] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:00.671049 8522 log.cc:1196] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/ccae5cec7fdf454d81282b311e948d59
I20240501 14:01:00.671598 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet ccae5cec7fdf454d81282b311e948d59 (table rebalancer_dynamic_table_new_name [id=85b95af3ee2c400b9775242e3dfd7dc5]) successfully deleted
I20240501 14:01:00.672168 8522 ts_tablet_manager.cc:1909] T ccae5cec7fdf454d81282b311e948d59 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting consensus metadata
I20240501 14:01:00.673267 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet ccae5cec7fdf454d81282b311e948d59 (table rebalancer_dynamic_table_new_name [id=85b95af3ee2c400b9775242e3dfd7dc5]) successfully deleted
I20240501 14:01:00.676219 8520 ts_tablet_manager.cc:1888] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:00.676330 8520 log.cc:1196] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/ccae5cec7fdf454d81282b311e948d59
I20240501 14:01:00.676656 8520 ts_tablet_manager.cc:1909] T ccae5cec7fdf454d81282b311e948d59 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:01:00.677704 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet ccae5cec7fdf454d81282b311e948d59 (table rebalancer_dynamic_table_new_name [id=85b95af3ee2c400b9775242e3dfd7dc5]) successfully deleted
I20240501 14:01:00.755405 2002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:00.764459 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0c32f7452546b6b5a1609e33091421 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]), partition=
I20240501 14:01:00.764849 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0c32f7452546b6b5a1609e33091421 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]), partition=
I20240501 14:01:00.764902 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0c32f7452546b6b5a1609e33091421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.765050 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0c32f7452546b6b5a1609e33091421 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]), partition=
I20240501 14:01:00.765390 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0c32f7452546b6b5a1609e33091421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.766314 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0c32f7452546b6b5a1609e33091421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.770869 8523 tablet_bootstrap.cc:492] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:00.773778 8523 tablet_bootstrap.cc:654] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.777781 8523 tablet_bootstrap.cc:492] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:00.778398 8523 ts_tablet_manager.cc:1367] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240501 14:01:00.779532 8523 raft_consensus.cc:348] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.780558 8523 raft_consensus.cc:374] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.781026 8523 raft_consensus.cc:729] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.781564 8523 consensus_queue.cc:260] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.782219 8526 tablet_bootstrap.cc:492] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:00.782634 8524 tablet_bootstrap.cc:492] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:01:00.783607 8523 ts_tablet_manager.cc:1398] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:00.784519 8524 tablet_bootstrap.cc:654] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.785478 8526 tablet_bootstrap.cc:654] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.786739 8524 tablet_bootstrap.cc:492] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:01:00.786903 8524 ts_tablet_manager.cc:1367] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:01:00.787133 8526 tablet_bootstrap.cc:492] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:00.787313 8526 ts_tablet_manager.cc:1367] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:01:00.788038 8526 raft_consensus.cc:348] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.788309 8526 raft_consensus.cc:374] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.788417 8526 raft_consensus.cc:729] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.788705 8526 consensus_queue.cc:260] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.789299 8526 ts_tablet_manager.cc:1398] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:00.789722 8524 raft_consensus.cc:348] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.789891 8524 raft_consensus.cc:374] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.789988 8524 raft_consensus.cc:729] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.790182 8524 consensus_queue.cc:260] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.790823 8524 ts_tablet_manager.cc:1398] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:00.805990 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:00.816946 8496 raft_consensus.cc:482] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:00.817131 8496 raft_consensus.cc:504] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.817927 8496 leader_election.cc:288] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:00.818353 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0c32f7452546b6b5a1609e33091421" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:01:00.818616 5031 raft_consensus.cc:2454] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:00.819195 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0c32f7452546b6b5a1609e33091421" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:00.819428 2408 raft_consensus.cc:2454] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:00.819845 5097 leader_election.cc:302] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:00.820132 8496 raft_consensus.cc:2781] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:00.820248 8496 raft_consensus.cc:482] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:00.820331 8496 raft_consensus.cc:3037] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.822266 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 91184ecf7ec3409ca3b05318aa5dcab3 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.822870 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91184ecf7ec3409ca3b05318aa5dcab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.823173 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 0e407965bc6047b185913b10dd0681fa (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.823166 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 909f45a4783b4a1e95d3afb9c3f0b72b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.823529 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909f45a4783b4a1e95d3afb9c3f0b72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.823580 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e407965bc6047b185913b10dd0681fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.826980 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 909f45a4783b4a1e95d3afb9c3f0b72b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.827486 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909f45a4783b4a1e95d3afb9c3f0b72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.828572 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 91184ecf7ec3409ca3b05318aa5dcab3 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.828989 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91184ecf7ec3409ca3b05318aa5dcab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.829754 8496 raft_consensus.cc:504] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.831228 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 91184ecf7ec3409ca3b05318aa5dcab3 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.831501 8496 leader_election.cc:288] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:00.831635 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91184ecf7ec3409ca3b05318aa5dcab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.832695 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 0e407965bc6047b185913b10dd0681fa (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.833065 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e407965bc6047b185913b10dd0681fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.833977 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0c32f7452546b6b5a1609e33091421" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:00.834199 2408 raft_consensus.cc:3037] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.836735 2408 raft_consensus.cc:2454] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:00.837365 5097 leader_election.cc:302] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:00.832634 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 909f45a4783b4a1e95d3afb9c3f0b72b (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.837847 8514 raft_consensus.cc:2781] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:00.837996 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909f45a4783b4a1e95d3afb9c3f0b72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.838019 8514 raft_consensus.cc:686] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:01:00.838335 8514 consensus_queue.cc:237] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:00.842836 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0c32f7452546b6b5a1609e33091421" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:01:00.843098 5032 raft_consensus.cc:3037] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:00.843073 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 0e407965bc6047b185913b10dd0681fa (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:00.843531 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e407965bc6047b185913b10dd0681fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:00.845178 7872 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.846218 5032 raft_consensus.cc:2454] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:00.849701 7872 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:00.870735 8523 tablet_bootstrap.cc:492] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:00.872396 8523 tablet_bootstrap.cc:654] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.891345 8523 tablet_bootstrap.cc:492] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:00.891638 8523 ts_tablet_manager.cc:1367] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240501 14:01:00.892134 8530 tablet_bootstrap.cc:492] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:00.892797 8523 raft_consensus.cc:348] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:00.893761 8523 raft_consensus.cc:374] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.893858 8523 raft_consensus.cc:729] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.894105 8523 consensus_queue.cc:260] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:00.894677 8523 ts_tablet_manager.cc:1398] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:00.895272 8537 tablet_bootstrap.cc:492] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:00.897298 8538 tablet_bootstrap.cc:492] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:00.897564 8536 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:00.900894 8530 tablet_bootstrap.cc:654] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.902297 8537 tablet_bootstrap.cc:654] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.902814 8538 tablet_bootstrap.cc:654] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.903046 8536 tablet_bootstrap.cc:654] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.905251 8535 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:00.906904 1997 catalog_manager.cc:5484] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.919415 8537 tablet_bootstrap.cc:492] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:00.919675 8535 tablet_bootstrap.cc:654] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.919831 8537 ts_tablet_manager.cc:1367] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240501 14:01:00.921994 8537 raft_consensus.cc:348] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.922297 8537 raft_consensus.cc:374] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.922428 8537 raft_consensus.cc:729] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.922724 8537 consensus_queue.cc:260] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.927296 8537 ts_tablet_manager.cc:1398] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:00.932299 8537 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:00.932875 8538 tablet_bootstrap.cc:492] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:01:00.933709 8536 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:01:00.933946 8536 ts_tablet_manager.cc:1367] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20240501 14:01:00.934795 8536 raft_consensus.cc:348] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.935210 8538 ts_tablet_manager.cc:1367] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.006s
I20240501 14:01:00.935348 8537 tablet_bootstrap.cc:654] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.935977 8538 raft_consensus.cc:348] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.936204 8538 raft_consensus.cc:374] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.938410 8536 raft_consensus.cc:374] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.938645 8536 raft_consensus.cc:729] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.938894 8536 consensus_queue.cc:260] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.936267 8538 raft_consensus.cc:729] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.939388 8538 consensus_queue.cc:260] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.939730 8537 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:00.940042 8538 ts_tablet_manager.cc:1398] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 14:01:00.940066 8537 ts_tablet_manager.cc:1367] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:01:00.942337 8536 ts_tablet_manager.cc:1398] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:01:00.942706 8536 tablet_bootstrap.cc:492] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:00.943596 8530 tablet_bootstrap.cc:492] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:01:00.944061 8530 ts_tablet_manager.cc:1367] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.012s
I20240501 14:01:00.945444 8530 raft_consensus.cc:348] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:00.945720 8530 raft_consensus.cc:374] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.945816 8530 raft_consensus.cc:729] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.946062 8530 consensus_queue.cc:260] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:00.946934 8535 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:01:00.947134 8535 ts_tablet_manager.cc:1367] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20240501 14:01:00.947841 8535 raft_consensus.cc:348] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.948094 8535 raft_consensus.cc:374] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.948161 8535 raft_consensus.cc:729] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.948387 8535 consensus_queue.cc:260] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.949103 8535 ts_tablet_manager.cc:1398] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:00.949893 8536 tablet_bootstrap.cc:654] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.957317 8530 ts_tablet_manager.cc:1398] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240501 14:01:00.960314 8536 tablet_bootstrap.cc:492] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:01:00.960602 8536 ts_tablet_manager.cc:1367] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.005s
I20240501 14:01:00.961568 8536 raft_consensus.cc:348] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.961881 8536 raft_consensus.cc:374] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.962018 8536 raft_consensus.cc:729] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.962433 8536 consensus_queue.cc:260] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:00.964433 8536 ts_tablet_manager.cc:1398] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:00.965840 8537 raft_consensus.cc:348] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.966169 8537 raft_consensus.cc:374] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.966305 8537 raft_consensus.cc:729] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.966676 8537 consensus_queue.cc:260] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:00.974145 8537 ts_tablet_manager.cc:1398] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240501 14:01:00.974694 8537 tablet_bootstrap.cc:492] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:00.981647 8537 tablet_bootstrap.cc:654] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:00.984138 8537 tablet_bootstrap.cc:492] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:00.984344 8537 ts_tablet_manager.cc:1367] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240501 14:01:00.984961 8537 raft_consensus.cc:348] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:00.985138 8537 raft_consensus.cc:374] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:00.985224 8537 raft_consensus.cc:729] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:00.985450 8537 consensus_queue.cc:260] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:00.987545 8537 ts_tablet_manager.cc:1398] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:01:01.010479 8462 raft_consensus.cc:482] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.010710 8462 raft_consensus.cc:504] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.011449 8462 leader_election.cc:288] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:01.011822 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91184ecf7ec3409ca3b05318aa5dcab3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:01.012060 5296 raft_consensus.cc:2454] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:01:01.012029 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91184ecf7ec3409ca3b05318aa5dcab3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:01:01.012257 4899 raft_consensus.cc:2454] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:01:01.012483 2213 leader_election.cc:302] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:01.012807 8462 raft_consensus.cc:2781] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.012900 8462 raft_consensus.cc:482] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.012986 8462 raft_consensus.cc:3037] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.015138 8462 raft_consensus.cc:504] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.016383 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91184ecf7ec3409ca3b05318aa5dcab3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:01.016400 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91184ecf7ec3409ca3b05318aa5dcab3" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:01:01.016620 5296 raft_consensus.cc:3037] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.016619 4899 raft_consensus.cc:3037] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.019464 4899 raft_consensus.cc:2454] T 91184ecf7ec3409ca3b05318aa5dcab3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:01:01.019464 5296 raft_consensus.cc:2454] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:01:01.019888 2212 leader_election.cc:302] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:01.015861 8462 leader_election.cc:288] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:01.020190 8462 raft_consensus.cc:2781] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.020332 8462 raft_consensus.cc:686] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:01:01.020613 8462 consensus_queue.cc:237] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.024659 1997 catalog_manager.cc:5484] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:01.026433 2002 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:01.030977 8307 raft_consensus.cc:482] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.031147 8307 raft_consensus.cc:504] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:01.032137 8307 leader_election.cc:288] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:01.032548 5032 raft_consensus.cc:1261] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:01.032964 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909f45a4783b4a1e95d3afb9c3f0b72b" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:01.033201 5296 raft_consensus.cc:2454] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:01:01.033407 2406 raft_consensus.cc:1261] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:01.033677 2473 leader_election.cc:302] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters:
I20240501 14:01:01.033941 8307 raft_consensus.cc:2781] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.034135 8307 raft_consensus.cc:482] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.034202 8307 raft_consensus.cc:3037] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.035122 8514 consensus_queue.cc:1035] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.035406 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909f45a4783b4a1e95d3afb9c3f0b72b" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:01.035619 5161 raft_consensus.cc:2454] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:01:01.035650 8534 consensus_queue.cc:1035] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.036674 8307 raft_consensus.cc:504] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:01.037549 8307 leader_election.cc:288] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:01.038072 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909f45a4783b4a1e95d3afb9c3f0b72b" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:01.038331 5296 raft_consensus.cc:3037] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.038700 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909f45a4783b4a1e95d3afb9c3f0b72b" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:01.038928 5161 raft_consensus.cc:3037] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.041299 5296 raft_consensus.cc:2454] T 909f45a4783b4a1e95d3afb9c3f0b72b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:01.042601 2002 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]). Expected version 1 got 0
I20240501 14:01:01.044782 8552 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.045331 8552 tablet.cc:1620] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:01:01.047201 5161 raft_consensus.cc:2454] T 909f45a4783b4a1e95d3afb9c3f0b72b P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:01.048528 1997 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]). Expected version 1 got 0
I20240501 14:01:01.049249 2002 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]). Expected version 1 got 0
I20240501 14:01:01.049610 2471 leader_election.cc:302] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:01.050185 8161 raft_consensus.cc:2781] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.050297 8161 raft_consensus.cc:686] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:01:01.050566 8161 consensus_queue.cc:237] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:01.051043 8554 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.051648 8554 tablet.cc:1620] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:01:01.055835 2002 catalog_manager.cc:5576] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037) does not have the latest schema for tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]). Expected version 1 got 0
I20240501 14:01:01.065471 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5] alter complete (version 1)
I20240501 14:01:01.067241 8551 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.070457 8551 tablet.cc:1620] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:01:01.070524 2002 catalog_manager.cc:5484] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:01.073164 1997 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5]). Expected version 1 got 0
I20240501 14:01:01.077823 8385 raft_consensus.cc:482] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.078013 8385 raft_consensus.cc:504] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:01.078822 8385 leader_election.cc:288] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:01.079380 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e407965bc6047b185913b10dd0681fa" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:01:01.079370 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e407965bc6047b185913b10dd0681fa" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:01.079588 4899 raft_consensus.cc:2454] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:01:01.079599 5296 raft_consensus.cc:2454] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:01:01.079900 2081 leader_election.cc:302] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:01.080158 8385 raft_consensus.cc:2781] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.080231 8385 raft_consensus.cc:482] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.080294 8385 raft_consensus.cc:3037] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.082888 8385 raft_consensus.cc:504] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:01.083446 8385 leader_election.cc:288] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:01.083848 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e407965bc6047b185913b10dd0681fa" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:01:01.083886 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e407965bc6047b185913b10dd0681fa" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:01.084054 4899 raft_consensus.cc:3037] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.084163 5296 raft_consensus.cc:3037] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.086548 4899 raft_consensus.cc:2454] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:01.086884 5296 raft_consensus.cc:2454] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:01.087100 2081 leader_election.cc:302] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:01.087368 8385 raft_consensus.cc:2781] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.087496 8385 raft_consensus.cc:686] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:01.087755 8385 consensus_queue.cc:237] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:01.091699 2005 catalog_manager.cc:5484] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:01.176213 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:01.183674 8551 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:01.184593 8551 tablet.cc:1620] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:01:01.186848 8554 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:01.187446 8554 tablet.cc:1620] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: MemRowSet was empty: no flush needed.
I20240501 14:01:01.191124 8552 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:01.191596 8552 tablet.cc:1620] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:01:01.195288 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=6998af06e7f443d2bf6b3e49ffeaeff5] alter complete (version 2)
I20240501 14:01:01.296761 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:01.304351 8551 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:01.305487 8552 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:01.305943 8554 tablet.cc:1722] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:01.315979 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=6998af06e7f443d2bf6b3e49ffeaeff5] alter complete (version 3)
I20240501 14:01:01.420634 2002 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:01.420874 2002 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:01.424015 2002 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 0e0c32f7452546b6b5a1609e33091421
I20240501 14:01:01.425038 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e0c32f7452546b6b5a1609e33091421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:01 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:01.425218 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e0c32f7452546b6b5a1609e33091421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:01 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:01:01.425621 8557 tablet_replica.cc:330] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:01.425680 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e0c32f7452546b6b5a1609e33091421 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:01 UTC) from {username='slave'} at 127.0.0.1:58280
I20240501 14:01:01.425937 8557 raft_consensus.cc:2229] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:01.426353 8557 raft_consensus.cc:2258] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.426365 8559 tablet_replica.cc:330] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:01:01.426770 8559 raft_consensus.cc:2229] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:01.427116 8559 raft_consensus.cc:2258] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.427253 8557 ts_tablet_manager.cc:1875] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:01.428025 8559 ts_tablet_manager.cc:1875] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:01.434309 8557 ts_tablet_manager.cc:1888] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:01.434540 8557 log.cc:1196] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/0e0c32f7452546b6b5a1609e33091421
I20240501 14:01:01.435011 8557 ts_tablet_manager.cc:1909] T 0e0c32f7452546b6b5a1609e33091421 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:01:01.436170 8510 consensus_queue.cc:1035] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.436304 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table_new_name [id=6998af06e7f443d2bf6b3e49ffeaeff5]) successfully deleted
I20240501 14:01:01.437273 8559 ts_tablet_manager.cc:1888] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:01.437402 8559 log.cc:1196] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/0e0c32f7452546b6b5a1609e33091421
I20240501 14:01:01.437762 8559 ts_tablet_manager.cc:1909] T 0e0c32f7452546b6b5a1609e33091421 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting consensus metadata
I20240501 14:01:01.438760 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table_new_name [id=6998af06e7f443d2bf6b3e49ffeaeff5]) successfully deleted
I20240501 14:01:01.438843 8558 tablet_replica.cc:330] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:01:01.439152 8558 raft_consensus.cc:2229] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:01.439376 8558 raft_consensus.cc:2258] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.440171 8558 ts_tablet_manager.cc:1875] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:01.446288 8558 ts_tablet_manager.cc:1888] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:01.446465 8558 log.cc:1196] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e0c32f7452546b6b5a1609e33091421
I20240501 14:01:01.446521 8510 consensus_queue.cc:1035] T 909f45a4783b4a1e95d3afb9c3f0b72b P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.446926 8558 ts_tablet_manager.cc:1909] T 0e0c32f7452546b6b5a1609e33091421 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:01:01.448547 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet 0e0c32f7452546b6b5a1609e33091421 (table rebalancer_dynamic_table_new_name [id=6998af06e7f443d2bf6b3e49ffeaeff5]) successfully deleted
I20240501 14:01:01.526580 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:01.534152 5275 tablet_service.cc:1467] Processing CreateTablet for tablet ec1744dea8324e16a9b5d177c4b14824 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]), partition=
I20240501 14:01:01.534240 2387 tablet_service.cc:1467] Processing CreateTablet for tablet ec1744dea8324e16a9b5d177c4b14824 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]), partition=
I20240501 14:01:01.534611 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1744dea8324e16a9b5d177c4b14824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.534615 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1744dea8324e16a9b5d177c4b14824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.534829 5144 tablet_service.cc:1467] Processing CreateTablet for tablet ec1744dea8324e16a9b5d177c4b14824 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]), partition=
I20240501 14:01:01.535210 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1744dea8324e16a9b5d177c4b14824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.539880 8385 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.540812 8564 tablet_bootstrap.cc:492] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:01.543381 8566 tablet_bootstrap.cc:492] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:01.542330 8564 tablet_bootstrap.cc:654] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.545166 8566 tablet_bootstrap.cc:654] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.551760 8462 consensus_queue.cc:1035] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.570703 8567 tablet_bootstrap.cc:492] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:01.571640 8564 tablet_bootstrap.cc:492] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:01.571832 8564 ts_tablet_manager.cc:1367] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240501 14:01:01.572439 8567 tablet_bootstrap.cc:654] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.573100 8462 consensus_queue.cc:1035] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.575987 8385 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.601578 8567 tablet_bootstrap.cc:492] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:01.601778 8567 ts_tablet_manager.cc:1367] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.008s
I20240501 14:01:01.602691 8567 raft_consensus.cc:348] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.603047 8567 raft_consensus.cc:374] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.603228 8567 raft_consensus.cc:729] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.604754 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:01.607903 8567 consensus_queue.cc:260] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.609719 8567 ts_tablet_manager.cc:1398] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:01.616128 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 67da9784350e49209e979e232a79cfc1 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.616596 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67da9784350e49209e979e232a79cfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.618392 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 2e20cd9040e4448b93ac513371a7fcf9 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.618868 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e20cd9040e4448b93ac513371a7fcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.621949 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 14dc7387f47942cfa5e96aeac0805a5f (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.622362 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dc7387f47942cfa5e96aeac0805a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.622320 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 14dc7387f47942cfa5e96aeac0805a5f (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.622722 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dc7387f47942cfa5e96aeac0805a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.623773 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 14dc7387f47942cfa5e96aeac0805a5f (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.624145 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14dc7387f47942cfa5e96aeac0805a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.625793 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 2e20cd9040e4448b93ac513371a7fcf9 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.626181 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e20cd9040e4448b93ac513371a7fcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.631008 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 67da9784350e49209e979e232a79cfc1 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.631489 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67da9784350e49209e979e232a79cfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.634164 8564 raft_consensus.cc:348] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.634377 8564 raft_consensus.cc:374] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.634419 8564 raft_consensus.cc:729] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.634667 8564 consensus_queue.cc:260] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.635293 8564 ts_tablet_manager.cc:1398] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.063s user 0.002s sys 0.000s
I20240501 14:01:01.643393 8580 tablet_bootstrap.cc:492] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:01.648392 8580 tablet_bootstrap.cc:654] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.651458 8578 tablet_bootstrap.cc:492] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:01.652565 8580 tablet_bootstrap.cc:492] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:01:01.652702 8580 ts_tablet_manager.cc:1367] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:01:01.655014 8578 tablet_bootstrap.cc:654] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.655359 8580 raft_consensus.cc:348] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.655627 8580 raft_consensus.cc:374] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.655699 8580 raft_consensus.cc:729] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.657289 8580 consensus_queue.cc:260] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.660599 8580 ts_tablet_manager.cc:1398] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:01:01.665784 8564 tablet_bootstrap.cc:492] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:01.667366 8564 tablet_bootstrap.cc:654] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.672942 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 67da9784350e49209e979e232a79cfc1 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.673449 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67da9784350e49209e979e232a79cfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.679054 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 2e20cd9040e4448b93ac513371a7fcf9 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=86e1eb79babe4732a2bc21f2d8c6dd56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:01.679589 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e20cd9040e4448b93ac513371a7fcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.690754 8564 tablet_bootstrap.cc:492] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:01.691002 8564 ts_tablet_manager.cc:1367] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240501 14:01:01.690093 8584 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:01:01.691903 8564 raft_consensus.cc:348] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.692071 8583 tablet_bootstrap.cc:492] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:01.692185 8564 raft_consensus.cc:374] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.692337 8564 raft_consensus.cc:729] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.692651 8564 consensus_queue.cc:260] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.697906 8583 tablet_bootstrap.cc:654] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.698165 8564 ts_tablet_manager.cc:1398] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:01.698606 8564 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:01.699340 8584 tablet_bootstrap.cc:654] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.704855 8588 tablet_bootstrap.cc:492] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:01.707279 8566 tablet_bootstrap.cc:492] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:01.707589 8566 ts_tablet_manager.cc:1367] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.165s user 0.039s sys 0.024s
I20240501 14:01:01.708361 8588 tablet_bootstrap.cc:654] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.710309 8584 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:01:01.710575 8584 ts_tablet_manager.cc:1367] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 14:01:01.710587 8566 raft_consensus.cc:348] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.710945 8566 raft_consensus.cc:374] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.711823 8564 tablet_bootstrap.cc:654] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.712404 8566 raft_consensus.cc:729] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.712703 8566 consensus_queue.cc:260] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.713513 8566 ts_tablet_manager.cc:1398] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:01.713749 8584 raft_consensus.cc:348] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.714007 8584 raft_consensus.cc:374] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.714097 8584 raft_consensus.cc:729] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.715361 8588 tablet_bootstrap.cc:492] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:01:01.715369 8578 tablet_bootstrap.cc:492] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:01:01.715530 8578 ts_tablet_manager.cc:1367] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.000s
I20240501 14:01:01.715634 8588 ts_tablet_manager.cc:1367] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:01:01.716436 8578 raft_consensus.cc:348] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.716454 8588 raft_consensus.cc:348] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.716663 8578 raft_consensus.cc:374] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.716751 8578 raft_consensus.cc:729] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.716801 8588 raft_consensus.cc:374] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.716972 8588 raft_consensus.cc:729] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.717034 8578 consensus_queue.cc:260] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.717310 8588 consensus_queue.cc:260] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.717996 8588 ts_tablet_manager.cc:1398] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:01.718297 8583 tablet_bootstrap.cc:492] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:01:01.718327 8588 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:01.718557 8583 ts_tablet_manager.cc:1367] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
W20240501 14:01:01.718801 2211 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:41309, user_credentials={real_user=slave}} blocked reactor thread for 45430.1us
I20240501 14:01:01.719600 8566 tablet_bootstrap.cc:492] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:01.719810 8583 raft_consensus.cc:348] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.720458 8578 ts_tablet_manager.cc:1398] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:01.720880 8584 consensus_queue.cc:260] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.723030 8588 tablet_bootstrap.cc:654] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.724720 8584 ts_tablet_manager.cc:1398] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:01:01.728593 8583 raft_consensus.cc:374] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.728771 8583 raft_consensus.cc:729] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.729089 8583 consensus_queue.cc:260] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.732606 8496 raft_consensus.cc:482] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.732769 8496 raft_consensus.cc:504] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.733489 8496 leader_election.cc:288] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:01.734136 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1744dea8324e16a9b5d177c4b14824" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:01.734344 8588 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:01:01.734377 5296 raft_consensus.cc:2454] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:01.734529 8588 ts_tablet_manager.cc:1367] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240501 14:01:01.735340 8588 raft_consensus.cc:348] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.735553 8588 raft_consensus.cc:374] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.735617 8588 raft_consensus.cc:729] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.735823 8588 consensus_queue.cc:260] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.736392 8588 ts_tablet_manager.cc:1398] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:01.736439 8566 tablet_bootstrap.cc:654] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:01.737360 5099 leader_election.cc:302] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:01.737637 8496 raft_consensus.cc:2781] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.737740 8496 raft_consensus.cc:482] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.737811 8496 raft_consensus.cc:3037] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.738173 8564 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:01.738317 8564 ts_tablet_manager.cc:1367] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.003s
I20240501 14:01:01.738927 8564 raft_consensus.cc:348] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.739063 8564 raft_consensus.cc:374] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.739092 8564 raft_consensus.cc:729] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.739260 8564 consensus_queue.cc:260] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.739781 8564 ts_tablet_manager.cc:1398] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:01:01.740597 8496 raft_consensus.cc:504] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.742339 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1744dea8324e16a9b5d177c4b14824" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:01.742537 2406 raft_consensus.cc:3037] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.745319 2406 raft_consensus.cc:2454] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:01.745965 5097 leader_election.cc:302] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:01.746201 8566 tablet_bootstrap.cc:492] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:01.746265 8534 raft_consensus.cc:2781] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.746390 8566 ts_tablet_manager.cc:1367] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240501 14:01:01.747596 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1744dea8324e16a9b5d177c4b14824" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:01.747824 5296 raft_consensus.cc:3037] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.748361 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1744dea8324e16a9b5d177c4b14824" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:01.748543 2408 raft_consensus.cc:2362] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1. Re-sending same reply.
I20240501 14:01:01.751348 5296 raft_consensus.cc:2454] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:01.753002 8387 raft_consensus.cc:482] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.753187 8387 raft_consensus.cc:504] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.754110 8387 leader_election.cc:288] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:01.754578 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:01.753156 8566 raft_consensus.cc:348] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.754776 2408 raft_consensus.cc:2454] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:01:01.754829 8566 raft_consensus.cc:374] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:01.754902 8566 raft_consensus.cc:729] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:01.755321 4967 leader_election.cc:302] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:01.755227 8566 consensus_queue.cc:260] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.756645 8387 raft_consensus.cc:2781] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.756763 8387 raft_consensus.cc:482] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.756866 8387 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.760696 8496 leader_election.cc:288] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:01.762482 8387 raft_consensus.cc:504] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.763186 8387 leader_election.cc:288] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:01.763072 8566 ts_tablet_manager.cc:1398] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.017s user 0.002s sys 0.005s
I20240501 14:01:01.763653 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:01.763849 2408 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.765897 2408 raft_consensus.cc:2454] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:01:01.766323 4967 leader_election.cc:302] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:01.766961 8579 raft_consensus.cc:2781] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.767130 8579 raft_consensus.cc:686] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:01:01.767448 8579 consensus_queue.cc:237] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:01.769325 8583 ts_tablet_manager.cc:1398] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20240501 14:01:01.771158 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:01:01.771387 2274 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.772348 2000 catalog_manager.cc:5484] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:01.773576 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:01.773707 2274 raft_consensus.cc:2454] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:01:01.777853 8534 raft_consensus.cc:686] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:01:01.778218 8534 consensus_queue.cc:237] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.787241 2000 catalog_manager.cc:5484] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:01.818699 8480 raft_consensus.cc:482] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.818966 8480 raft_consensus.cc:504] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.819859 8480 leader_election.cc:288] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:01.820533 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e20cd9040e4448b93ac513371a7fcf9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:01:01.820747 2145 raft_consensus.cc:2454] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:01:01.821592 4834 leader_election.cc:302] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:01.821910 8480 raft_consensus.cc:2781] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.821864 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e20cd9040e4448b93ac513371a7fcf9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:01.822077 2277 raft_consensus.cc:2454] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:01:01.822093 8480 raft_consensus.cc:482] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.822177 8480 raft_consensus.cc:3037] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.825848 8480 raft_consensus.cc:504] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.827450 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e20cd9040e4448b93ac513371a7fcf9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:01:01.827665 2145 raft_consensus.cc:3037] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.828589 8480 leader_election.cc:288] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:01.830397 2145 raft_consensus.cc:2454] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:01.832201 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e20cd9040e4448b93ac513371a7fcf9" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:01:01.832443 2274 raft_consensus.cc:3037] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.834849 4834 leader_election.cc:302] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:01.837015 2274 raft_consensus.cc:2454] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:01.837692 8480 raft_consensus.cc:2781] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.837949 8480 raft_consensus.cc:686] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:01.838253 8480 consensus_queue.cc:237] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:01.842487 2000 catalog_manager.cc:5484] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:01.869599 8307 raft_consensus.cc:482] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:01.869776 8307 raft_consensus.cc:504] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.870458 8307 leader_election.cc:288] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:01.870990 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dc7387f47942cfa5e96aeac0805a5f" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:01.871217 2408 raft_consensus.cc:2454] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:01:01.871248 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dc7387f47942cfa5e96aeac0805a5f" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:01.871446 5164 raft_consensus.cc:2454] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:01:01.871646 2471 leader_election.cc:302] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:01.871896 8307 raft_consensus.cc:2781] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:01.871978 8307 raft_consensus.cc:482] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:01.872068 8307 raft_consensus.cc:3037] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.874066 8307 raft_consensus.cc:504] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.874702 8307 leader_election.cc:288] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:01.875151 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dc7387f47942cfa5e96aeac0805a5f" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:01.875357 2408 raft_consensus.cc:3037] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.875334 5164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14dc7387f47942cfa5e96aeac0805a5f" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:01.875540 5164 raft_consensus.cc:3037] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:01.877244 2408 raft_consensus.cc:2454] T 14dc7387f47942cfa5e96aeac0805a5f P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:01.877744 2471 leader_election.cc:302] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:01.878135 8307 raft_consensus.cc:2781] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:01.878260 8307 raft_consensus.cc:686] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:01:01.878351 5164 raft_consensus.cc:2454] T 14dc7387f47942cfa5e96aeac0805a5f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:01.878547 8307 consensus_queue.cc:237] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:01.885490 2000 catalog_manager.cc:5484] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:01.927449 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:01.934228 5296 raft_consensus.cc:1261] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:01.934566 2408 raft_consensus.cc:1261] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:01.935128 8596 consensus_queue.cc:1035] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.935432 8534 consensus_queue.cc:1035] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:01.941622 1997 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]). Expected version 1 got 0
I20240501 14:01:01.942328 2000 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]). Expected version 1 got 0
I20240501 14:01:01.943665 8607 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.946429 8607 tablet.cc:1620] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:01:01.950299 2000 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]). Expected version 1 got 0
I20240501 14:01:01.951751 8608 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.952522 8608 tablet.cc:1620] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:01:01.955529 8606 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:01.956210 2000 catalog_manager.cc:5576] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537) does not have the latest schema for tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]). Expected version 1 got 0
I20240501 14:01:01.959741 2000 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]). Expected version 1 got 0
I20240501 14:01:01.964646 8606 tablet.cc:1620] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:01:01.964800 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83] alter complete (version 1)
I20240501 14:01:01.965850 2005 catalog_manager.cc:5576] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601) does not have the latest schema for tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83]). Expected version 1 got 0
I20240501 14:01:02.069509 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:02.077443 8606 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.078015 8606 tablet.cc:1620] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: MemRowSet was empty: no flush needed.
I20240501 14:01:02.078730 8608 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.077919 8607 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.079448 8608 tablet.cc:1620] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: MemRowSet was empty: no flush needed.
I20240501 14:01:02.079887 8607 tablet.cc:1620] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:01:02.092016 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=77736e6cef9d479eaa7c28d7dd00ec83] alter complete (version 2)
I20240501 14:01:02.193953 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:02.202673 8606 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:02.203230 8608 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:02.205101 8607 tablet.cc:1722] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:02.214008 1997 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=77736e6cef9d479eaa7c28d7dd00ec83] alter complete (version 3)
I20240501 14:01:02.299229 8387 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.302294 8480 consensus_queue.cc:1035] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.312254 8611 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 14:01:02.329034 2005 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:02.329279 2005 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:02.334549 2005 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet ec1744dea8324e16a9b5d177c4b14824
I20240501 14:01:02.335438 5275 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1744dea8324e16a9b5d177c4b14824 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:02 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:01:02.337375 8617 consensus_queue.cc:1035] T 2e20cd9040e4448b93ac513371a7fcf9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.336719 8618 tablet_replica.cc:330] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:01:02.338620 2387 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1744dea8324e16a9b5d177c4b14824 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:02 UTC) from {username='slave'} at 127.0.0.1:51622
I20240501 14:01:02.338687 8618 raft_consensus.cc:2229] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.338915 8618 raft_consensus.cc:2258] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.340463 8618 ts_tablet_manager.cc:1875] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:02.344306 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1744dea8324e16a9b5d177c4b14824 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:02 UTC) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:02.348284 8618 ts_tablet_manager.cc:1888] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:02.348562 8618 log.cc:1196] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/ec1744dea8324e16a9b5d177c4b14824
I20240501 14:01:02.349151 8618 ts_tablet_manager.cc:1909] T ec1744dea8324e16a9b5d177c4b14824 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:01:02.350775 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table_new_name [id=77736e6cef9d479eaa7c28d7dd00ec83]) successfully deleted
I20240501 14:01:02.357801 8620 tablet_replica.cc:330] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:01:02.358791 8620 raft_consensus.cc:2229] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:02.359294 8620 raft_consensus.cc:2258] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.359889 8622 tablet_replica.cc:330] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:02.360247 8620 ts_tablet_manager.cc:1875] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:02.360234 8622 raft_consensus.cc:2229] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:02.360695 8622 raft_consensus.cc:2258] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:02.361794 8622 ts_tablet_manager.cc:1875] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:02.367074 8620 ts_tablet_manager.cc:1888] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:02.367261 8620 log.cc:1196] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/ec1744dea8324e16a9b5d177c4b14824
I20240501 14:01:02.367672 8620 ts_tablet_manager.cc:1909] T ec1744dea8324e16a9b5d177c4b14824 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting consensus metadata
I20240501 14:01:02.368849 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table_new_name [id=77736e6cef9d479eaa7c28d7dd00ec83]) successfully deleted
I20240501 14:01:02.373152 8622 ts_tablet_manager.cc:1888] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:02.373284 8622 log.cc:1196] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/ec1744dea8324e16a9b5d177c4b14824
I20240501 14:01:02.373780 8622 ts_tablet_manager.cc:1909] T ec1744dea8324e16a9b5d177c4b14824 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting consensus metadata
I20240501 14:01:02.374949 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet ec1744dea8324e16a9b5d177c4b14824 (table rebalancer_dynamic_table_new_name [id=77736e6cef9d479eaa7c28d7dd00ec83]) successfully deleted
I20240501 14:01:02.390833 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:02.401980 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 20cd7aaca5434c2dbb63e7111feed3a9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.401958 2517 tablet_service.cc:1467] Processing CreateTablet for tablet 20cd7aaca5434c2dbb63e7111feed3a9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.402467 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cd7aaca5434c2dbb63e7111feed3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.402467 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cd7aaca5434c2dbb63e7111feed3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.402784 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4312eec255417580288818bd559099 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.403159 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4312eec255417580288818bd559099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.407611 5014 tablet_service.cc:1467] Processing CreateTablet for tablet 20cd7aaca5434c2dbb63e7111feed3a9 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.407981 5275 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4312eec255417580288818bd559099 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.408111 5014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20cd7aaca5434c2dbb63e7111feed3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.408382 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4312eec255417580288818bd559099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.407611 5013 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4312eec255417580288818bd559099 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.411068 5013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4312eec255417580288818bd559099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.422948 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 926fd4668e8644969365d57749be3e1f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.423254 5012 tablet_service.cc:1467] Processing CreateTablet for tablet 926fd4668e8644969365d57749be3e1f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.423494 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926fd4668e8644969365d57749be3e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.423631 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926fd4668e8644969365d57749be3e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.423939 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 926fd4668e8644969365d57749be3e1f (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=079eb75a791d4340a2d5419aad89afaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:02.424352 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926fd4668e8644969365d57749be3e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.439246 8629 tablet_bootstrap.cc:492] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:02.441814 8629 tablet_bootstrap.cc:654] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.444283 8628 tablet_bootstrap.cc:492] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:02.445022 8627 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:01:02.447340 1997 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:02.454309 8307 consensus_queue.cc:1035] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.457456 8629 tablet_bootstrap.cc:492] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:01:02.457650 8629 ts_tablet_manager.cc:1367] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 14:01:02.458480 8629 raft_consensus.cc:348] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.458738 8629 raft_consensus.cc:374] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.458817 8629 raft_consensus.cc:729] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.459046 8629 consensus_queue.cc:260] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.459609 8629 ts_tablet_manager.cc:1398] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:02.469264 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 746dba2acb5147c2b44116f2fd0b283b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279]), partition=
I20240501 14:01:02.469715 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746dba2acb5147c2b44116f2fd0b283b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.473853 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 746dba2acb5147c2b44116f2fd0b283b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279]), partition=
I20240501 14:01:02.474242 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746dba2acb5147c2b44116f2fd0b283b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.477315 8626 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:02.478302 8510 consensus_queue.cc:1035] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.478936 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 746dba2acb5147c2b44116f2fd0b283b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279]), partition=
I20240501 14:01:02.479296 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746dba2acb5147c2b44116f2fd0b283b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.483599 8626 tablet_bootstrap.cc:654] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.493932 8624 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:02.496727 8624 tablet_bootstrap.cc:654] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.503082 8625 tablet_bootstrap.cc:492] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:02.508883 8626 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:02.509192 8626 ts_tablet_manager.cc:1367] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 14:01:02.510282 8626 raft_consensus.cc:348] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.510623 8626 raft_consensus.cc:374] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.510777 8626 raft_consensus.cc:729] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.511121 8626 consensus_queue.cc:260] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.514536 8624 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:01:02.516616 8626 ts_tablet_manager.cc:1398] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:02.517020 8626 tablet_bootstrap.cc:492] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:02.517328 8624 ts_tablet_manager.cc:1367] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240501 14:01:02.519078 8626 tablet_bootstrap.cc:654] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.519867 8624 raft_consensus.cc:348] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.524555 8625 tablet_bootstrap.cc:654] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.524730 8624 raft_consensus.cc:374] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.524880 8624 raft_consensus.cc:729] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.525337 8624 consensus_queue.cc:260] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.529076 8624 ts_tablet_manager.cc:1398] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:01:02.530618 8624 tablet_bootstrap.cc:492] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:02.532121 8624 tablet_bootstrap.cc:654] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.547056 8627 tablet_bootstrap.cc:654] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.548853 8628 tablet_bootstrap.cc:654] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.550002 8625 tablet_bootstrap.cc:492] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:01:02.550196 8625 ts_tablet_manager.cc:1367] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.000s
I20240501 14:01:02.550987 8625 raft_consensus.cc:348] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.551220 8625 raft_consensus.cc:374] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.552354 8630 tablet_bootstrap.cc:492] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:02.552793 8625 raft_consensus.cc:729] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.553506 8625 consensus_queue.cc:260] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.555308 8624 tablet_bootstrap.cc:492] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:01:02.555773 8625 ts_tablet_manager.cc:1398] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:01:02.555980 8624 ts_tablet_manager.cc:1367] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240501 14:01:02.556280 8625 tablet_bootstrap.cc:492] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:02.556967 8624 raft_consensus.cc:348] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.558461 8625 tablet_bootstrap.cc:654] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.561658 8627 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:01:02.561838 8630 tablet_bootstrap.cc:654] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.562574 8624 raft_consensus.cc:374] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.562670 8624 raft_consensus.cc:729] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.562876 8624 consensus_queue.cc:260] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.563138 8627 ts_tablet_manager.cc:1367] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.000s
I20240501 14:01:02.564432 8624 ts_tablet_manager.cc:1398] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:01:02.566066 8627 raft_consensus.cc:348] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.566296 8627 raft_consensus.cc:374] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.566900 8627 raft_consensus.cc:729] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.567624 8625 tablet_bootstrap.cc:492] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:01:02.568291 8626 tablet_bootstrap.cc:492] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:02.568516 8626 ts_tablet_manager.cc:1367] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.000s
I20240501 14:01:02.569211 8625 ts_tablet_manager.cc:1367] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240501 14:01:02.569628 8630 tablet_bootstrap.cc:492] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:02.569581 8626 raft_consensus.cc:348] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.569833 8626 raft_consensus.cc:374] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.569929 8626 raft_consensus.cc:729] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.570036 8625 raft_consensus.cc:348] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.570184 8626 consensus_queue.cc:260] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.570313 8625 raft_consensus.cc:374] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.570770 8630 ts_tablet_manager.cc:1367] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240501 14:01:02.570906 8626 ts_tablet_manager.cc:1398] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:02.571250 8627 consensus_queue.cc:260] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.571620 8509 raft_consensus.cc:482] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.570376 8625 raft_consensus.cc:729] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.571754 8509 raft_consensus.cc:504] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.572481 8509 leader_election.cc:288] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:02.572422 8625 consensus_queue.cc:260] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.572948 8625 ts_tablet_manager.cc:1398] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:01:02.574697 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746dba2acb5147c2b44116f2fd0b283b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:02.574901 2274 raft_consensus.cc:2454] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:01:02.575477 5228 leader_election.cc:302] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:02.575788 8509 raft_consensus.cc:2781] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:02.575901 8509 raft_consensus.cc:482] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:02.575989 8509 raft_consensus.cc:3037] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.574769 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746dba2acb5147c2b44116f2fd0b283b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:02.578649 8509 raft_consensus.cc:504] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.578763 2537 raft_consensus.cc:2454] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 0.
I20240501 14:01:02.579707 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746dba2acb5147c2b44116f2fd0b283b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:02.579880 2537 raft_consensus.cc:3037] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.580113 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746dba2acb5147c2b44116f2fd0b283b" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:01:02.580291 2274 raft_consensus.cc:3037] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.582787 2274 raft_consensus.cc:2454] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:01:02.585057 5228 leader_election.cc:302] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:02.585443 8411 raft_consensus.cc:2781] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:02.587800 8509 leader_election.cc:288] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:02.588642 2537 raft_consensus.cc:2454] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:01:02.592777 8462 raft_consensus.cc:482] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.592936 8462 raft_consensus.cc:504] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.593511 8630 raft_consensus.cc:348] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.593712 8630 raft_consensus.cc:374] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.593703 8462 leader_election.cc:288] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:02.593760 8630 raft_consensus.cc:729] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.594787 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4312eec255417580288818bd559099" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
I20240501 14:01:02.595036 8628 tablet_bootstrap.cc:492] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:01:02.595041 5032 raft_consensus.cc:2454] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:01:02.595176 8628 ts_tablet_manager.cc:1367] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.000s
I20240501 14:01:02.595455 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4312eec255417580288818bd559099" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:02.595665 5296 raft_consensus.cc:2454] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 0.
I20240501 14:01:02.595978 8628 raft_consensus.cc:348] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.596165 8628 raft_consensus.cc:374] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.596240 8628 raft_consensus.cc:729] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.596307 2213 leader_election.cc:302] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:02.596583 8462 raft_consensus.cc:2781] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:02.596673 8462 raft_consensus.cc:482] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:02.596546 8628 consensus_queue.cc:260] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.596729 8462 raft_consensus.cc:3037] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.599735 8462 raft_consensus.cc:504] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.599956 8628 ts_tablet_manager.cc:1398] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:02.600513 8462 leader_election.cc:288] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:02.600665 8609 raft_consensus.cc:482] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.600869 8609 raft_consensus.cc:504] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.601070 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4312eec255417580288818bd559099" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:02.601083 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4312eec255417580288818bd559099" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:01:02.601248 5296 raft_consensus.cc:3037] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.601279 5032 raft_consensus.cc:3037] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.601542 8609 leader_election.cc:288] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:02.602203 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cd7aaca5434c2dbb63e7111feed3a9" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:02.602423 2537 raft_consensus.cc:2454] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:01:02.602854 2080 leader_election.cc:302] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:02.603112 8609 raft_consensus.cc:2781] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:02.603201 8609 raft_consensus.cc:482] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:02.603271 8609 raft_consensus.cc:3037] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.603909 5296 raft_consensus.cc:2454] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:01:02.604154 8627 ts_tablet_manager.cc:1398] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20240501 14:01:02.604175 5032 raft_consensus.cc:2454] T 5c4312eec255417580288818bd559099 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
I20240501 14:01:02.605451 8627 tablet_bootstrap.cc:492] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:01:02.604943 8630 consensus_queue.cc:260] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.605629 8609 raft_consensus.cc:504] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.606559 8609 leader_election.cc:288] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:02.606655 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cd7aaca5434c2dbb63e7111feed3a9" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:02.606745 5032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cd7aaca5434c2dbb63e7111feed3a9" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2"
I20240501 14:01:02.606832 2537 raft_consensus.cc:3037] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:01:02.607151 2082 leader_election.cc:341] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:02.604972 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cd7aaca5434c2dbb63e7111feed3a9" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" is_pre_election: true
W20240501 14:01:02.608227 2082 leader_election.cc:341] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:02.609457 2537 raft_consensus.cc:2454] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:02.609799 2080 leader_election.cc:302] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:02.610025 8609 raft_consensus.cc:2781] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:02.610239 8609 raft_consensus.cc:686] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:02.610563 8609 consensus_queue.cc:237] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.612636 2213 leader_election.cc:302] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:02.613147 8462 raft_consensus.cc:2781] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:02.613276 8462 raft_consensus.cc:686] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:01:02.613520 8462 consensus_queue.cc:237] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:02.615918 2000 catalog_manager.cc:5484] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:02.621059 8630 ts_tablet_manager.cc:1398] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240501 14:01:02.621717 2000 catalog_manager.cc:5484] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 0 to 1, leader changed from <none> to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:02.624831 8627 tablet_bootstrap.cc:654] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.626763 8411 raft_consensus.cc:686] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:01:02.627079 8411 consensus_queue.cc:237] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:02.628037 8627 tablet_bootstrap.cc:492] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:01:02.628252 8627 ts_tablet_manager.cc:1367] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 14:01:02.629300 8627 raft_consensus.cc:348] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.629552 8627 raft_consensus.cc:374] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.629652 8627 raft_consensus.cc:729] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.629906 8627 consensus_queue.cc:260] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.630681 8627 ts_tablet_manager.cc:1398] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:02.631263 8627 tablet_bootstrap.cc:492] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:01:02.632969 8627 tablet_bootstrap.cc:654] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:02.634135 2002 catalog_manager.cc:5484] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:02.649690 8627 tablet_bootstrap.cc:492] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2: No bootstrap required, opened a new log
I20240501 14:01:02.650290 8627 ts_tablet_manager.cc:1367] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20240501 14:01:02.651522 8627 raft_consensus.cc:348] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.651950 8627 raft_consensus.cc:374] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:02.652123 8627 raft_consensus.cc:729] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: FOLLOWER
I20240501 14:01:02.652359 8627 consensus_queue.cc:260] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:02.653045 8627 ts_tablet_manager.cc:1398] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:02.674557 8611 raft_consensus.cc:482] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:02.674763 8611 raft_consensus.cc:504] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.675455 8611 leader_election.cc:288] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:02.675858 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926fd4668e8644969365d57749be3e1f" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:01:02.676122 4899 raft_consensus.cc:2454] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:01:02.676946 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926fd4668e8644969365d57749be3e1f" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:02.677160 2406 raft_consensus.cc:2454] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 0.
I20240501 14:01:02.677618 4967 leader_election.cc:302] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:02.677992 8611 raft_consensus.cc:2781] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:02.678073 8611 raft_consensus.cc:482] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:02.678157 8611 raft_consensus.cc:3037] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.684626 8611 raft_consensus.cc:504] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.685915 8611 leader_election.cc:288] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:02.685875 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926fd4668e8644969365d57749be3e1f" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:01:02.685972 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926fd4668e8644969365d57749be3e1f" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:02.686143 2406 raft_consensus.cc:3037] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.686134 4899 raft_consensus.cc:3037] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:02.688956 2406 raft_consensus.cc:2454] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:01:02.688977 4899 raft_consensus.cc:2454] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in term 1.
I20240501 14:01:02.690066 4967 leader_election.cc:302] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:02.690395 8611 raft_consensus.cc:2781] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:02.691208 8611 raft_consensus.cc:686] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Running, Role: LEADER
I20240501 14:01:02.691610 8611 consensus_queue.cc:237] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:02.696517 2005 catalog_manager.cc:5484] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6). New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:02.767747 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:02.775637 2537 raft_consensus.cc:1261] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.776262 8411 consensus_queue.cc:1035] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.776913 2274 raft_consensus.cc:1261] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:02.777562 8411 consensus_queue.cc:1035] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:02.782879 2005 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet 746dba2acb5147c2b44116f2fd0b283b (table rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279]). Expected version 1 got 0
I20240501 14:01:02.784610 8656 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:02.785123 8656 tablet.cc:1620] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:01:02.788745 2000 catalog_manager.cc:5576] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301) does not have the latest schema for tablet 746dba2acb5147c2b44116f2fd0b283b (table rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279]). Expected version 1 got 0
I20240501 14:01:02.795132 8655 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:02.795943 8655 tablet.cc:1620] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:01:02.800650 2002 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet 746dba2acb5147c2b44116f2fd0b283b (table rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279]). Expected version 1 got 0
I20240501 14:01:02.804519 2000 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279] alter complete (version 1)
I20240501 14:01:02.818593 8657 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:02.819746 8657 tablet.cc:1620] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:01:02.823849 2002 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet 746dba2acb5147c2b44116f2fd0b283b (table rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279]). Expected version 1 got 0
I20240501 14:01:02.928355 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:02.937389 8655 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.937901 8655 tablet.cc:1620] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:01:02.940416 8656 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.940912 8656 tablet.cc:1620] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: MemRowSet was empty: no flush needed.
I20240501 14:01:02.942695 8657 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:02.943117 8657 tablet.cc:1620] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:01:02.950935 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=dfbf76b664f44dc1b9ebba598677a279] alter complete (version 2)
I20240501 14:01:03.022621 8462 consensus_queue.cc:1035] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.028564 8663 consensus_queue.cc:1035] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.048661 8660 raft_consensus.cc:482] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.048844 8660 raft_consensus.cc:504] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:03.049773 8660 leader_election.cc:288] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:03.050318 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cd7aaca5434c2dbb63e7111feed3a9" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:03.050344 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20cd7aaca5434c2dbb63e7111feed3a9" candidate_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:01:03.050608 2538 raft_consensus.cc:2379] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f81f803ec7746ccb502c59b7cbf8eb2 in current term 1: Already voted for candidate d1a414d381f44829a2510cb76d481887 in this term.
I20240501 14:01:03.051199 4967 leader_election.cc:302] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f81f803ec7746ccb502c59b7cbf8eb2; no voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887
I20240501 14:01:03.051589 8660 raft_consensus.cc:3037] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.054226 8660 raft_consensus.cc:2726] T 20cd7aaca5434c2dbb63e7111feed3a9 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:03.055245 2000 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:01:03.064519 8655 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:03.065232 8657 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:03.066910 8656 tablet.cc:1722] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:01:03.077492 2002 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table_new_name [id=dfbf76b664f44dc1b9ebba598677a279] alter complete (version 3)
I20240501 14:01:03.078696 8609 consensus_queue.cc:1035] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.099216 8385 consensus_queue.cc:1035] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.101485 8660 consensus_queue.cc:1035] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.106889 8387 consensus_queue.cc:1035] T 926fd4668e8644969365d57749be3e1f P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.182610 2005 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:03.182852 2005 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:01:03.185954 2005 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: Sending DeleteTablet for 3 replicas of tablet 746dba2acb5147c2b44116f2fd0b283b
I20240501 14:01:03.187103 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 746dba2acb5147c2b44116f2fd0b283b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:03 UTC) from {username='slave'} at 127.0.0.1:44554
I20240501 14:01:03.187100 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 746dba2acb5147c2b44116f2fd0b283b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:03 UTC) from {username='slave'} at 127.0.0.1:36154
I20240501 14:01:03.187222 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 746dba2acb5147c2b44116f2fd0b283b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:01:03 UTC) from {username='slave'} at 127.0.0.1:35676
I20240501 14:01:03.187875 8675 tablet_replica.cc:330] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:01:03.188112 8675 raft_consensus.cc:2229] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:03.188627 8675 raft_consensus.cc:2258] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:03.190784 8675 ts_tablet_manager.cc:1875] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:03.199800 8675 ts_tablet_manager.cc:1888] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:03.200119 8675 log.cc:1196] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/746dba2acb5147c2b44116f2fd0b283b
I20240501 14:01:03.201937 8675 ts_tablet_manager.cc:1909] T 746dba2acb5147c2b44116f2fd0b283b P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting consensus metadata
I20240501 14:01:03.203672 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 746dba2acb5147c2b44116f2fd0b283b (table rebalancer_dynamic_table_new_name [id=dfbf76b664f44dc1b9ebba598677a279]) successfully deleted
I20240501 14:01:03.203747 8677 tablet_replica.cc:330] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:01:03.203907 8676 tablet_replica.cc:330] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:01:03.203976 8677 raft_consensus.cc:2229] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:03.204108 8676 raft_consensus.cc:2229] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:03.204278 8677 raft_consensus.cc:2258] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:03.204368 8676 raft_consensus.cc:2258] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:03.205278 8677 ts_tablet_manager.cc:1875] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:03.207087 8676 ts_tablet_manager.cc:1875] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:01:03.210860 8677 ts_tablet_manager.cc:1888] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:03.211062 8677 log.cc:1196] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/746dba2acb5147c2b44116f2fd0b283b
I20240501 14:01:03.211472 8677 ts_tablet_manager.cc:1909] T 746dba2acb5147c2b44116f2fd0b283b P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting consensus metadata
I20240501 14:01:03.212585 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 746dba2acb5147c2b44116f2fd0b283b (table rebalancer_dynamic_table_new_name [id=dfbf76b664f44dc1b9ebba598677a279]) successfully deleted
I20240501 14:01:03.213076 8676 ts_tablet_manager.cc:1888] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:01:03.213187 8676 log.cc:1196] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/746dba2acb5147c2b44116f2fd0b283b
I20240501 14:01:03.213523 8676 ts_tablet_manager.cc:1909] T 746dba2acb5147c2b44116f2fd0b283b P 745ac12076e84effa5bde1ecbd6a32ec: Deleting consensus metadata
I20240501 14:01:03.214488 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 746dba2acb5147c2b44116f2fd0b283b (table rebalancer_dynamic_table_new_name [id=dfbf76b664f44dc1b9ebba598677a279]) successfully deleted
I20240501 14:01:03.239701 2005 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:03.248813 5144 tablet_service.cc:1467] Processing CreateTablet for tablet fdb2924e91fb4a98aaa6bf977f682fdb (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.248868 2387 tablet_service.cc:1467] Processing CreateTablet for tablet fdb2924e91fb4a98aaa6bf977f682fdb (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.249265 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb2924e91fb4a98aaa6bf977f682fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.249274 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb2924e91fb4a98aaa6bf977f682fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.254969 2258 tablet_service.cc:1467] Processing CreateTablet for tablet fdb2924e91fb4a98aaa6bf977f682fdb (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.255427 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb2924e91fb4a98aaa6bf977f682fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.256774 4879 tablet_service.cc:1467] Processing CreateTablet for tablet b9ce2e195d834f6b986ac7273e7c6b78 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.257226 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ce2e195d834f6b986ac7273e7c6b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.257753 2517 tablet_service.cc:1467] Processing CreateTablet for tablet b9ce2e195d834f6b986ac7273e7c6b78 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.257848 8679 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:03.258121 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ce2e195d834f6b986ac7273e7c6b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.258939 4881 tablet_service.cc:1467] Processing CreateTablet for tablet 8502afe4b8a7439b882961145c9475f5 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.259361 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8502afe4b8a7439b882961145c9475f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.259620 8679 tablet_bootstrap.cc:654] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.262763 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 8502afe4b8a7439b882961145c9475f5 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.263216 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8502afe4b8a7439b882961145c9475f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.267177 2388 tablet_service.cc:1467] Processing CreateTablet for tablet b9ce2e195d834f6b986ac7273e7c6b78 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.267647 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ce2e195d834f6b986ac7273e7c6b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.270961 8679 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:03.271306 8679 ts_tablet_manager.cc:1367] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240501 14:01:03.272819 8679 raft_consensus.cc:348] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.273264 8679 raft_consensus.cc:374] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.273454 8679 raft_consensus.cc:729] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.273752 8679 consensus_queue.cc:260] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.274447 8679 ts_tablet_manager.cc:1398] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:01:03.277086 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 8502afe4b8a7439b882961145c9475f5 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:03.277518 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8502afe4b8a7439b882961145c9475f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.291368 2000 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34122:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:01:03.298213 8682 tablet_bootstrap.cc:492] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:03.301936 8682 tablet_bootstrap.cc:654] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.303609 8684 tablet_bootstrap.cc:492] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:03.305632 8684 tablet_bootstrap.cc:654] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.310549 2125 tablet_service.cc:1467] Processing CreateTablet for tablet 11a9e4400b4a4a4694e7449e902828c0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]), partition=
I20240501 14:01:03.310601 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 11a9e4400b4a4a4694e7449e902828c0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]), partition=
I20240501 14:01:03.310953 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a9e4400b4a4a4694e7449e902828c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.310969 2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a9e4400b4a4a4694e7449e902828c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.324460 8685 tablet_bootstrap.cc:492] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:03.327091 8678 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:03.327903 8685 tablet_bootstrap.cc:654] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.328974 8678 tablet_bootstrap.cc:654] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.331681 8684 tablet_bootstrap.cc:492] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:01:03.331841 8684 ts_tablet_manager.cc:1367] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240501 14:01:03.332639 8684 raft_consensus.cc:348] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.332938 8684 raft_consensus.cc:374] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.333091 8684 raft_consensus.cc:729] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.333259 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 11a9e4400b4a4a4694e7449e902828c0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]), partition=
I20240501 14:01:03.333405 8684 consensus_queue.cc:260] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.333644 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a9e4400b4a4a4694e7449e902828c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:03.334080 8684 ts_tablet_manager.cc:1398] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:03.334360 8684 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:03.336524 8678 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:03.336675 8678 ts_tablet_manager.cc:1367] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 14:01:03.337471 8678 raft_consensus.cc:348] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.337685 8678 raft_consensus.cc:374] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.337771 8678 raft_consensus.cc:729] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.337989 8678 consensus_queue.cc:260] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.338578 8678 ts_tablet_manager.cc:1398] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:03.339521 8682 tablet_bootstrap.cc:492] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:03.339679 8682 ts_tablet_manager.cc:1367] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.002s
I20240501 14:01:03.340498 8679 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:03.342110 8679 tablet_bootstrap.cc:654] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.342680 8685 tablet_bootstrap.cc:492] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: No bootstrap required, opened a new log
I20240501 14:01:03.343544 8685 ts_tablet_manager.cc:1367] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240501 14:01:03.343832 8682 raft_consensus.cc:348] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.344156 8682 raft_consensus.cc:374] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.344300 8682 raft_consensus.cc:729] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.344506 8679 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd: No bootstrap required, opened a new log
I20240501 14:01:03.344664 8679 ts_tablet_manager.cc:1367] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:03.345417 8679 raft_consensus.cc:348] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.345626 8679 raft_consensus.cc:374] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.345710 8679 raft_consensus.cc:729] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.345561 8682 consensus_queue.cc:260] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.345934 8679 consensus_queue.cc:260] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.346487 8679 ts_tablet_manager.cc:1398] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:03.347640 8685 raft_consensus.cc:348] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.347862 8685 raft_consensus.cc:374] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.348340 8682 ts_tablet_manager.cc:1398] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240501 14:01:03.348865 8678 tablet_bootstrap.cc:492] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:03.349030 8685 raft_consensus.cc:729] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.349529 8685 consensus_queue.cc:260] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.350679 8685 ts_tablet_manager.cc:1398] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:03.351620 8682 tablet_bootstrap.cc:492] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:03.355818 8682 tablet_bootstrap.cc:654] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.359511 8684 tablet_bootstrap.cc:654] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.361179 8678 tablet_bootstrap.cc:654] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.385053 8659 raft_consensus.cc:482] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.385282 8659 raft_consensus.cc:504] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.386097 8659 leader_election.cc:288] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:03.387167 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb2924e91fb4a98aaa6bf977f682fdb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:03.387904 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb2924e91fb4a98aaa6bf977f682fdb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
W20240501 14:01:03.388015 5097 leader_election.cc:341] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:03.388191 2408 raft_consensus.cc:2454] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 0.
I20240501 14:01:03.388900 5097 leader_election.cc:302] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:03.389499 8659 raft_consensus.cc:2781] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.389637 8659 raft_consensus.cc:482] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.389722 8659 raft_consensus.cc:3037] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.395481 8659 raft_consensus.cc:504] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.397186 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb2924e91fb4a98aaa6bf977f682fdb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
W20240501 14:01:03.397907 5097 leader_election.cc:341] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:03.398200 8659 leader_election.cc:288] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:03.399075 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb2924e91fb4a98aaa6bf977f682fdb" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:03.399333 2408 raft_consensus.cc:3037] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.402468 2408 raft_consensus.cc:2454] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:03.409111 5097 leader_election.cc:302] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:03.411583 8658 raft_consensus.cc:2781] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.413599 8658 raft_consensus.cc:686] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:01:03.414011 8658 consensus_queue.cc:237] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.430337 8689 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:03.432699 8689 tablet_bootstrap.cc:654] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.443425 2005 catalog_manager.cc:5484] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:03.446153 8681 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:03.449815 8684 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: No bootstrap required, opened a new log
I20240501 14:01:03.450101 8684 ts_tablet_manager.cc:1367] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.116s user 0.009s sys 0.020s
I20240501 14:01:03.451455 8684 raft_consensus.cc:348] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.453109 8684 raft_consensus.cc:374] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.453261 8684 raft_consensus.cc:729] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.453590 8684 consensus_queue.cc:260] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.458545 8681 tablet_bootstrap.cc:654] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.458611 8480 raft_consensus.cc:482] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.458782 8480 raft_consensus.cc:504] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.459175 8684 ts_tablet_manager.cc:1398] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 14:01:03.459612 8480 leader_election.cc:288] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:03.460309 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
W20240501 14:01:03.460850 4834 leader_election.cc:341] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:03.461397 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:03.461653 5296 raft_consensus.cc:2454] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 0.
I20240501 14:01:03.462536 4836 leader_election.cc:302] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:03.462828 8480 raft_consensus.cc:2781] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.462932 8480 raft_consensus.cc:482] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.463014 8480 raft_consensus.cc:3037] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.464383 8689 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba: No bootstrap required, opened a new log
I20240501 14:01:03.464511 8689 ts_tablet_manager.cc:1367] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240501 14:01:03.465291 8689 raft_consensus.cc:348] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.465523 8689 raft_consensus.cc:374] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.465569 8689 raft_consensus.cc:729] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.465597 8480 raft_consensus.cc:504] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.465776 8689 consensus_queue.cc:260] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.466708 8480 leader_election.cc:288] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:03.466733 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:03.466959 5296 raft_consensus.cc:3037] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.468327 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:03.469689 5296 raft_consensus.cc:2454] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
W20240501 14:01:03.469749 4834 leader_election.cc:341] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:01:03.470343 4836 leader_election.cc:302] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:03.470916 8480 raft_consensus.cc:2781] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.471057 8480 raft_consensus.cc:686] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:03.471195 8689 ts_tablet_manager.cc:1398] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:03.471316 8480 consensus_queue.cc:237] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.472949 8681 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:01:03.473115 8681 ts_tablet_manager.cc:1367] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240501 14:01:03.473315 8682 tablet_bootstrap.cc:492] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: No bootstrap required, opened a new log
I20240501 14:01:03.473651 8682 ts_tablet_manager.cc:1367] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.122s user 0.027s sys 0.038s
I20240501 14:01:03.473914 8681 raft_consensus.cc:348] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.474141 8681 raft_consensus.cc:374] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.474210 8681 raft_consensus.cc:729] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.474587 8682 raft_consensus.cc:348] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.474882 8682 raft_consensus.cc:374] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.475013 8682 raft_consensus.cc:729] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.477591 8678 tablet_bootstrap.cc:492] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd: No bootstrap required, opened a new log
I20240501 14:01:03.477458 8682 consensus_queue.cc:260] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.478302 8678 ts_tablet_manager.cc:1367] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.024s
I20240501 14:01:03.478331 8682 ts_tablet_manager.cc:1398] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:03.479611 8678 raft_consensus.cc:348] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.479815 8678 raft_consensus.cc:374] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.480901 8678 raft_consensus.cc:729] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.481343 8678 consensus_queue.cc:260] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.482362 8678 ts_tablet_manager.cc:1398] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:01:03.486397 2002 catalog_manager.cc:5484] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:03.474457 8681 consensus_queue.cc:260] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.498124 8681 ts_tablet_manager.cc:1398] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20240501 14:01:03.498559 8681 tablet_bootstrap.cc:492] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:03.500661 8681 tablet_bootstrap.cc:654] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:03.515692 8681 tablet_bootstrap.cc:492] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: No bootstrap required, opened a new log
I20240501 14:01:03.516502 8681 ts_tablet_manager.cc:1367] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20240501 14:01:03.517268 8681 raft_consensus.cc:348] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.517560 8681 raft_consensus.cc:374] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:03.517653 8681 raft_consensus.cc:729] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: FOLLOWER
I20240501 14:01:03.517895 8681 consensus_queue.cc:260] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.518602 8681 ts_tablet_manager.cc:1398] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:03.578387 8659 raft_consensus.cc:482] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.578584 8659 raft_consensus.cc:504] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:03.579329 8659 leader_election.cc:288] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:03.579835 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:01:03.581187 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:03.581471 5296 raft_consensus.cc:2379] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9f94ddc5ff64ad79b9f01846a20e2fd in current term 1: Already voted for candidate f8c60f2bbac541dcba4424ab726f9ad9 in this term.
I20240501 14:01:03.581972 5099 leader_election.cc:302] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:03.582638 8658 raft_consensus.cc:3037] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.584548 8658 raft_consensus.cc:2726] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:03.606464 8633 raft_consensus.cc:482] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.606694 8633 raft_consensus.cc:504] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.607578 8633 leader_election.cc:288] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:03.607766 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce2e195d834f6b986ac7273e7c6b78" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
I20240501 14:01:03.608016 4899 raft_consensus.cc:2454] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:01:03.608175 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce2e195d834f6b986ac7273e7c6b78" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:03.608386 2408 raft_consensus.cc:2454] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 0.
I20240501 14:01:03.608458 2472 leader_election.cc:302] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:03.608722 8633 raft_consensus.cc:2781] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.608824 8633 raft_consensus.cc:482] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.608927 8633 raft_consensus.cc:3037] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.611253 8633 raft_consensus.cc:504] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.611826 8633 leader_election.cc:288] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:03.612411 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce2e195d834f6b986ac7273e7c6b78" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:03.612601 2408 raft_consensus.cc:3037] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.612525 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce2e195d834f6b986ac7273e7c6b78" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:01:03.612746 4899 raft_consensus.cc:3037] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.614599 2408 raft_consensus.cc:2454] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:03.614790 4899 raft_consensus.cc:2454] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:03.614976 2471 leader_election.cc:302] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:03.615374 8633 raft_consensus.cc:2781] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.615515 8633 raft_consensus.cc:686] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:01:03.615778 8633 consensus_queue.cc:237] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:03.620151 1997 catalog_manager.cc:5484] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:03.715104 8385 raft_consensus.cc:482] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.715329 8385 raft_consensus.cc:504] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.715979 8385 leader_election.cc:288] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:03.716490 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a9e4400b4a4a4694e7449e902828c0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:03.716490 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a9e4400b4a4a4694e7449e902828c0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:03.716692 5293 raft_consensus.cc:2454] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:01:03.716706 2537 raft_consensus.cc:2454] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 0.
I20240501 14:01:03.717170 2080 leader_election.cc:302] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:03.717459 8385 raft_consensus.cc:2781] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:03.717556 8385 raft_consensus.cc:482] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:03.717641 8385 raft_consensus.cc:3037] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.719597 8385 raft_consensus.cc:504] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.720227 8385 leader_election.cc:288] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:03.720734 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a9e4400b4a4a4694e7449e902828c0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:03.720885 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a9e4400b4a4a4694e7449e902828c0" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:03.720958 5293 raft_consensus.cc:3037] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.721109 2537 raft_consensus.cc:3037] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.723690 5293 raft_consensus.cc:2454] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:03.723757 2537 raft_consensus.cc:2454] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:03.724152 2082 leader_election.cc:302] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:03.724478 8385 raft_consensus.cc:2781] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:03.724608 8385 raft_consensus.cc:686] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:03.724869 8385 consensus_queue.cc:237] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:03.728996 1997 catalog_manager.cc:5484] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:03.767927 8663 raft_consensus.cc:482] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:03.768100 8663 raft_consensus.cc:504] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:03.768769 8663 leader_election.cc:288] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:03.769169 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb2924e91fb4a98aaa6bf977f682fdb" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:03.769263 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb2924e91fb4a98aaa6bf977f682fdb" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:03.769501 2408 raft_consensus.cc:2379] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in current term 1: Already voted for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in this term.
I20240501 14:01:03.769775 2211 leader_election.cc:302] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:03.769995 8663 raft_consensus.cc:3037] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:03.771862 8663 raft_consensus.cc:2726] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:03.810029 8659 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.824616 8694 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.849439 8617 consensus_queue.cc:1035] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.861081 8619 consensus_queue.cc:1035] T 8502afe4b8a7439b882961145c9475f5 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.878285 1997 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:01:03.886420 5293 raft_consensus.cc:1261] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:03.886822 2537 raft_consensus.cc:1261] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:03.887574 8385 consensus_queue.cc:1035] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.888031 8385 consensus_queue.cc:1035] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:03.904701 1997 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 11a9e4400b4a4a4694e7449e902828c0 (table rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]). Expected version 1 got 0
I20240501 14:01:03.913290 2005 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet 11a9e4400b4a4a4694e7449e902828c0 (table rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]). Expected version 1 got 0
I20240501 14:01:03.920656 8717 tablet.cc:1722] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:03.921365 8717 tablet.cc:1620] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:01:03.923128 8720 tablet.cc:1722] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:03.923767 8720 tablet.cc:1620] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:01:03.927771 2005 catalog_manager.cc:5576] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309) does not have the latest schema for tablet 11a9e4400b4a4a4694e7449e902828c0 (table rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]). Expected version 1 got 0
I20240501 14:01:03.932713 1997 catalog_manager.cc:5576] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117) does not have the latest schema for tablet 11a9e4400b4a4a4694e7449e902828c0 (table rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]). Expected version 1 got 0
I20240501 14:01:03.934695 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f] alter complete (version 1)
I20240501 14:01:03.946630 1997 catalog_manager.cc:5576] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221) does not have the latest schema for tablet 11a9e4400b4a4a4694e7449e902828c0 (table rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f]). Expected version 1 got 0
I20240501 14:01:03.942390 8719 tablet.cc:1722] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:01:03.949601 8719 tablet.cc:1620] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:01:04.041887 2005 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34122:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:01:04.049252 8717 tablet.cc:1722] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:04.050027 8720 tablet.cc:1722] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:04.050642 8720 tablet.cc:1620] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec: MemRowSet was empty: no flush needed.
I20240501 14:01:04.050664 8719 tablet.cc:1722] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:01:04.051203 8719 tablet.cc:1620] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c: MemRowSet was empty: no flush needed.
I20240501 14:01:04.051213 8717 tablet.cc:1620] T 11a9e4400b4a4a4694e7449e902828c0 P d1a414d381f44829a2510cb76d481887: MemRowSet was empty: no flush needed.
I20240501 14:01:04.060863 2005 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 09c4cadf5e9843128fcc82f0a1415ab1: rebalancer_dynamic_table [id=794d72ad67294d48bb7c5992aed5867f] alter complete (version 2)
I20240501 14:01:04.107942 8307 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:04.114897 8307 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:05.044920 5013 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:05.046208 5276 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:05.050539 2517 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:01:05.141716 2388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:05.152848 4881 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:05.159827 5142 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:05.173079 2258 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:05.173530 2125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:05.752636 5013 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:05.761096 2517 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:01:05.773623 2388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:05.783516 4881 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:05.804560 5142 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:05.810075 5276 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:05.848181 2258 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:05.850633 2125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 14:01:05.946015 2341 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.2:46301, user_credentials={real_user=slave}} blocked reactor thread for 59190.8us
I20240501 14:01:06.398582 5013 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:06.423904 5276 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:06.424409 2517 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 14:01:06.459959 2388 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:06.477896 4881 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 14:01:06.498992 2258 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 14:01:06.579258 5142 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:01:06.588451 2125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:01:06.802796 5296 consensus_queue.cc:237] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:06.805398 2406 raft_consensus.cc:1261] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.805480 2538 raft_consensus.cc:1261] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.806020 8803 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:06.806350 5228 consensus_peers.cc:485] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bc3084368de4add8809f10451d96894. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:06.806700 8803 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:06.808959 8904 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:06.809820 2406 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:06.820969 2538 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:06.825467 2001 catalog_manager.cc:5484] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:06.863989 2538 consensus_queue.cc:237] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:06.868822 5034 raft_consensus.cc:1261] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:06.869518 2471 consensus_peers.cc:485] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6e71ec5dcbe4a08a0a7da5944dfe77d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:06.875138 8722 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:06.877017 5296 raft_consensus.cc:1261] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.877545 8722 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:06.881377 8954 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:06.882210 5034 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:06.888069 2006 catalog_manager.cc:5484] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:06.931545 5296 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:06.950615 5296 consensus_queue.cc:237] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:06.955633 2538 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.955675 4899 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.956243 8966 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:06.956629 5228 consensus_peers.cc:485] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9eec5cc39eda4f6d85f83959952a322c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:06.956962 8940 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:06.959417 8904 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:06.960688 4899 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:06.960880 2538 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:06.968325 2006 catalog_manager.cc:5484] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:06.988046 5034 consensus_queue.cc:237] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:06.990535 4899 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:06.991194 8611 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:06.991369 2274 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:06.991525 4967 consensus_peers.cc:485] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7b11568c2d7420fa06f903e52723583. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:06.991854 8611 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:06.994447 8976 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:06.995218 4899 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.000674 2001 catalog_manager.cc:5484] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.013633 2274 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.024006 5164 consensus_queue.cc:237] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:07.026774 2274 raft_consensus.cc:1261] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:07.027024 5097 consensus_peers.cc:485] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cdc95819b944c5590202e86cc955d91. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.027382 8694 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.028880 5296 raft_consensus.cc:1261] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.029525 8694 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.031630 8694 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:07.034229 5296 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:07.035815 2274 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:07.041848 2005 catalog_manager.cc:5484] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:07.061407 5164 consensus_queue.cc:237] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:07.064204 5034 raft_consensus.cc:1261] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.064292 2538 raft_consensus.cc:1261] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:07.064332 5099 consensus_peers.cc:485] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8e883d12de74534a02ca9ccaada086f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.064724 8988 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.065045 8694 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.067581 8694 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.068598 2538 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.069898 5034 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.074139 2006 catalog_manager.cc:5484] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.100502 2274 consensus_queue.cc:237] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
W20240501 14:01:07.103387 2212 consensus_peers.cc:485] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65306dbef405443fb4c7188ccb762b83. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.103659 2145 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:07.103783 5164 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:07.104204 8868 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:07.104462 9004 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:07.106989 8663 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:07.107837 5164 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:07.110910 2145 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:07.114918 2005 catalog_manager.cc:5484] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:07.147505 2274 consensus_queue.cc:237] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:07.149665 5164 raft_consensus.cc:1261] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.149715 5296 raft_consensus.cc:1261] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:07.149943 2211 consensus_peers.cc:485] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 878eb076eee94aa8988b3588d5395cba. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.150326 9004 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.150610 8868 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.153472 9004 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.154278 5296 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.155047 5164 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.160548 2001 catalog_manager.cc:5484] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.186784 2406 consensus_queue.cc:237] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:07.189159 5296 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:07.189244 2274 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:07.189744 8835 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:01:07.190109 2343 consensus_peers.cc:485] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e048df08cf4f7eaacb78d6592d79b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.190342 8835 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:07.193392 8835 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:07.195685 2274 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:07.197326 5293 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:07.201473 2005 catalog_manager.cc:5484] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.219997 2406 consensus_queue.cc:237] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:07.222066 2538 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:07.222066 5031 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:07.222647 8835 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:07.222925 8945 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:01:07.223258 2343 consensus_peers.cc:485] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c74bacbf4464bab9c403e2153953474. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.225569 8835 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.226239 2538 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.226414 5031 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.235060 2005 catalog_manager.cc:5484] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:07.250161 2538 consensus_queue.cc:237] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:07.252313 2146 raft_consensus.cc:1261] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.252360 5031 raft_consensus.cc:1261] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.253013 8957 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.253396 8722 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:07.253765 2472 consensus_peers.cc:485] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa75617adf64020af85f4cf2bd13917. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.256989 8722 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:07.257133 2146 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:07.260942 5031 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:07.268231 2001 catalog_manager.cc:5484] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.293078 5031 consensus_queue.cc:237] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:07.295511 5164 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.296161 8976 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:07.296551 4967 consensus_peers.cc:485] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea342bfcde3b4daa8dcc39126fd42cdc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.296649 4899 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.297230 8976 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.299361 8915 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.299913 5164 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.305050 4900 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.308082 2006 catalog_manager.cc:5484] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:07.327152 5164 consensus_queue.cc:237] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:07.338706 5031 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:07.339286 5293 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:01:07.339648 5097 consensus_peers.cc:485] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44a4e92b144f47b5800506f4bc3eadb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.339890 8985 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:07.340236 8988 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:07.343118 9056 ts_tablet_manager.cc:902] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:07.343994 8988 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:07.354825 5293 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:07.357177 2001 catalog_manager.cc:5484] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 4 to 5, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.378182 5034 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:07.393604 9056 tablet_copy_client.cc:308] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:01:07.396764 9066 ts_tablet_manager.cc:902] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:07.398362 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet 9eec5cc39eda4f6d85f83959952a322c from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:54649)
I20240501 14:01:07.398576 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet 9eec5cc39eda4f6d85f83959952a322c from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:54649: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-9eec5cc39eda4f6d85f83959952a322c
I20240501 14:01:07.401484 5316 tablet_copy_source_session.cc:205] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.404064 9056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eec5cc39eda4f6d85f83959952a322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.414714 9056 tablet_copy_client.cc:783] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.415158 9056 tablet_copy_client.cc:647] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.419019 9056 tablet_copy_client.cc:523] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.422132 9073 ts_tablet_manager.cc:902] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:07.422446 9056 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:07.430145 9056 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.430567 9056 tablet_bootstrap.cc:492] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:01:07.430837 9056 ts_tablet_manager.cc:1367] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.008s user 0.009s sys 0.000s
I20240501 14:01:07.431049 9072 ts_tablet_manager.cc:902] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:07.431914 9056 raft_consensus.cc:348] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:07.437269 9066 tablet_copy_client.cc:308] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.6:46037
I20240501 14:01:07.432260 9056 raft_consensus.cc:729] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:01:07.438004 5054 tablet_copy_service.cc:140] P 6f81f803ec7746ccb502c59b7cbf8eb2: Received BeginTabletCopySession request for tablet a7b11568c2d7420fa06f903e52723583 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:48677)
I20240501 14:01:07.438009 9056 consensus_queue.cc:260] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:07.438213 5054 tablet_copy_service.cc:161] P 6f81f803ec7746ccb502c59b7cbf8eb2: Beginning new tablet copy session on tablet a7b11568c2d7420fa06f903e52723583 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:48677: session id = a64f34785a7b44fa8070b0452a4e0ddd-a7b11568c2d7420fa06f903e52723583
I20240501 14:01:07.438853 9056 ts_tablet_manager.cc:1398] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:01:07.439914 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-9eec5cc39eda4f6d85f83959952a322c received from {username='slave'} at 127.28.17.7:54649
I20240501 14:01:07.440049 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-9eec5cc39eda4f6d85f83959952a322c on tablet 9eec5cc39eda4f6d85f83959952a322c with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:07.440672 9073 tablet_copy_client.cc:308] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:07.441463 5054 tablet_copy_source_session.cc:205] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.442800 9066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b11568c2d7420fa06f903e52723583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.443241 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d from peer d1a414d381f44829a2510cb76d481887 ({username='slave'} at 127.28.17.1:53971)
I20240501 14:01:07.443564 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet f6e71ec5dcbe4a08a0a7da5944dfe77d from peer d1a414d381f44829a2510cb76d481887 at {username='slave'} at 127.28.17.1:53971: session id = d1a414d381f44829a2510cb76d481887-f6e71ec5dcbe4a08a0a7da5944dfe77d
I20240501 14:01:07.449220 4900 consensus_queue.cc:237] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:07.450225 9066 tablet_copy_client.cc:783] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.450471 9066 tablet_copy_client.cc:647] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.450601 9072 tablet_copy_client.cc:308] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:01:07.452159 2558 tablet_copy_source_session.cc:205] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.452478 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet 3bc3084368de4add8809f10451d96894 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35917)
I20240501 14:01:07.452631 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet 3bc3084368de4add8809f10451d96894 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35917: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-3bc3084368de4add8809f10451d96894
I20240501 14:01:07.455144 5316 tablet_copy_source_session.cc:205] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.458993 9073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.460923 9066 tablet_copy_client.cc:523] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.461786 9072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc3084368de4add8809f10451d96894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.468523 9066 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:07.468848 9073 tablet_copy_client.cc:783] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.469110 9073 tablet_copy_client.cc:647] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 1 WAL segments...
W20240501 14:01:07.471259 4834 consensus_peers.cc:485] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9d4e5590d124918ac4814a86b006b69. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.471274 2146 raft_consensus.cc:1261] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.471985 8871 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.472177 5293 raft_consensus.cc:1261] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.472854 8871 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.476629 9073 tablet_copy_client.cc:523] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.480444 9073 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:07.481257 9072 tablet_copy_client.cc:783] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.481505 9072 tablet_copy_client.cc:647] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.488116 9072 tablet_copy_client.cc:523] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.490090 8871 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:07.490573 2145 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:07.491710 9072 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:07.498427 9066 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.498893 9066 tablet_bootstrap.cc:492] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:01:07.499131 9066 ts_tablet_manager.cc:1367] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.006s
I20240501 14:01:07.500015 9066 raft_consensus.cc:348] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:07.500331 9066 raft_consensus.cc:729] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:01:07.500291 2006 catalog_manager.cc:5484] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.500660 9066 consensus_queue.cc:260] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:07.501433 9066 ts_tablet_manager.cc:1398] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:07.504431 5054 tablet_copy_service.cc:342] P 6f81f803ec7746ccb502c59b7cbf8eb2: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-a7b11568c2d7420fa06f903e52723583 received from {username='slave'} at 127.28.17.3:48677
I20240501 14:01:07.504580 5054 tablet_copy_service.cc:433] P 6f81f803ec7746ccb502c59b7cbf8eb2: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-a7b11568c2d7420fa06f903e52723583 on tablet a7b11568c2d7420fa06f903e52723583 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:07.511868 9072 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.512244 9072 tablet_bootstrap.cc:492] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:07.512508 9072 ts_tablet_manager.cc:1367] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240501 14:01:07.513185 9072 raft_consensus.cc:348] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:07.513425 9072 raft_consensus.cc:729] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:07.513597 9072 consensus_queue.cc:260] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:07.514113 9072 ts_tablet_manager.cc:1398] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:07.514935 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-3bc3084368de4add8809f10451d96894 received from {username='slave'} at 127.28.17.2:35917
I20240501 14:01:07.515115 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-3bc3084368de4add8809f10451d96894 on tablet 3bc3084368de4add8809f10451d96894 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:07.529016 5293 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:07.534677 9093 ts_tablet_manager.cc:902] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:07.556977 9073 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.557382 9073 tablet_bootstrap.cc:492] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: Bootstrap complete.
I20240501 14:01:07.557803 9073 ts_tablet_manager.cc:1367] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.077s user 0.019s sys 0.015s
I20240501 14:01:07.558385 9073 raft_consensus.cc:348] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:07.558607 9073 raft_consensus.cc:729] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: LEARNER
I20240501 14:01:07.558746 9073 consensus_queue.cc:260] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:07.559264 9073 ts_tablet_manager.cc:1398] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:01:07.560672 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session d1a414d381f44829a2510cb76d481887-f6e71ec5dcbe4a08a0a7da5944dfe77d received from {username='slave'} at 127.28.17.1:53971
I20240501 14:01:07.560801 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session d1a414d381f44829a2510cb76d481887-f6e71ec5dcbe4a08a0a7da5944dfe77d on tablet f6e71ec5dcbe4a08a0a7da5944dfe77d with peer d1a414d381f44829a2510cb76d481887
I20240501 14:01:07.575973 9093 tablet_copy_client.cc:308] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:07.576751 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 8cdc95819b944c5590202e86cc955d91 from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:51047)
I20240501 14:01:07.576926 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 8cdc95819b944c5590202e86cc955d91 from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:51047: session id = 745ac12076e84effa5bde1ecbd6a32ec-8cdc95819b944c5590202e86cc955d91
I20240501 14:01:07.579953 5184 tablet_copy_source_session.cc:205] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.580133 2538 consensus_queue.cc:237] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:07.581377 9093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdc95819b944c5590202e86cc955d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.582784 4900 raft_consensus.cc:1261] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.583474 2406 raft_consensus.cc:1261] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.584530 8957 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.584939 8954 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:07.585850 2471 consensus_peers.cc:485] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9ce2e195d834f6b986ac7273e7c6b78. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.588830 9093 tablet_copy_client.cc:783] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.589130 9093 tablet_copy_client.cc:647] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.591102 9093 tablet_copy_client.cc:523] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.596518 9093 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:07.607427 8954 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:07.613157 9093 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.613519 9093 tablet_bootstrap.cc:492] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:07.613755 9093 ts_tablet_manager.cc:1367] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240501 14:01:07.614279 9093 raft_consensus.cc:348] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:07.614485 9093 raft_consensus.cc:729] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:07.614675 9093 consensus_queue.cc:260] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:07.615972 2406 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:07.616508 9093 ts_tablet_manager.cc:1398] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:01:07.617954 2006 catalog_manager.cc:5484] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.618885 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-8cdc95819b944c5590202e86cc955d91 received from {username='slave'} at 127.28.17.4:51047
I20240501 14:01:07.619027 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-8cdc95819b944c5590202e86cc955d91 on tablet 8cdc95819b944c5590202e86cc955d91 with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:07.623041 9107 ts_tablet_manager.cc:902] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:07.625715 4899 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:07.685938 9066 ts_tablet_manager.cc:902] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:07.722196 9119 ts_tablet_manager.cc:902] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:07.727766 9116 ts_tablet_manager.cc:902] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:07.729611 9111 ts_tablet_manager.cc:902] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:07.730715 9113 ts_tablet_manager.cc:902] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:07.733342 2408 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:07.734714 9107 tablet_copy_client.cc:308] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:07.735652 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet e8e883d12de74534a02ca9ccaada086f from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:50045)
I20240501 14:01:07.735837 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet e8e883d12de74534a02ca9ccaada086f from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:50045: session id = 5a4b9a96e9734baa8a03d6056bfa934c-e8e883d12de74534a02ca9ccaada086f
I20240501 14:01:07.736140 9066 tablet_copy_client.cc:308] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.6:46037
I20240501 14:01:07.736946 5054 tablet_copy_service.cc:140] P 6f81f803ec7746ccb502c59b7cbf8eb2: Received BeginTabletCopySession request for tablet ea342bfcde3b4daa8dcc39126fd42cdc from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:48677)
I20240501 14:01:07.737133 5054 tablet_copy_service.cc:161] P 6f81f803ec7746ccb502c59b7cbf8eb2: Beginning new tablet copy session on tablet ea342bfcde3b4daa8dcc39126fd42cdc from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:48677: session id = a64f34785a7b44fa8070b0452a4e0ddd-ea342bfcde3b4daa8dcc39126fd42cdc
I20240501 14:01:07.740072 5054 tablet_copy_source_session.cc:205] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.741628 9066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea342bfcde3b4daa8dcc39126fd42cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.742903 5184 tablet_copy_source_session.cc:205] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.744086 9107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e883d12de74534a02ca9ccaada086f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.750900 9072 ts_tablet_manager.cc:902] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:07.751343 9107 tablet_copy_client.cc:783] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.751722 9107 tablet_copy_client.cc:647] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.752139 9072 tablet_copy_client.cc:308] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:07.752430 9119 tablet_copy_client.cc:308] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:01:07.752827 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 44a4e92b144f47b5800506f4bc3eadb6 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:40205)
I20240501 14:01:07.753026 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 44a4e92b144f47b5800506f4bc3eadb6 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:40205: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-44a4e92b144f47b5800506f4bc3eadb6
I20240501 14:01:07.753247 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet f1e048df08cf4f7eaacb78d6592d79b1 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:41439)
I20240501 14:01:07.753401 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet f1e048df08cf4f7eaacb78d6592d79b1 from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:41439: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-f1e048df08cf4f7eaacb78d6592d79b1
I20240501 14:01:07.754303 9107 tablet_copy_client.cc:523] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.756306 5184 tablet_copy_source_session.cc:205] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.756318 2428 tablet_copy_source_session.cc:205] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.757758 9072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a4e92b144f47b5800506f4bc3eadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.760046 9119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e048df08cf4f7eaacb78d6592d79b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.763727 9066 tablet_copy_client.cc:783] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.763970 9066 tablet_copy_client.cc:647] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.764070 9107 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:07.768445 2146 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:07.772333 2341 consensus_peers.cc:485] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd3f8661a13944098a18d3eea7ddeb00. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.772851 8835 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:07.773373 4899 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:07.773936 9113 tablet_copy_client.cc:308] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:01:07.775334 8835 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:07.776645 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 65306dbef405443fb4c7188ccb762b83 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:52743)
I20240501 14:01:07.776705 9066 tablet_copy_client.cc:523] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.776784 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 65306dbef405443fb4c7188ccb762b83 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:52743: session id = f8c60f2bbac541dcba4424ab726f9ad9-65306dbef405443fb4c7188ccb762b83
I20240501 14:01:07.779809 2298 tablet_copy_source_session.cc:205] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.780208 9066 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:07.781221 9072 tablet_copy_client.cc:783] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.781601 9072 tablet_copy_client.cc:647] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.782548 9113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65306dbef405443fb4c7188ccb762b83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.784626 9072 tablet_copy_client.cc:523] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.787737 9111 tablet_copy_client.cc:308] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:01:07.788544 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet 6c74bacbf4464bab9c403e2153953474 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:53689)
I20240501 14:01:07.788707 9072 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:07.788702 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet 6c74bacbf4464bab9c403e2153953474 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:53689: session id = 5a4b9a96e9734baa8a03d6056bfa934c-6c74bacbf4464bab9c403e2153953474
I20240501 14:01:07.791623 2428 tablet_copy_source_session.cc:205] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.793010 9111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c74bacbf4464bab9c403e2153953474. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.795594 2274 raft_consensus.cc:1203] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:07.803174 8835 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:07.806468 9066 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.806895 9066 tablet_bootstrap.cc:492] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:01:07.807155 9066 ts_tablet_manager.cc:1367] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.001s
I20240501 14:01:07.809834 9107 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.810324 9107 tablet_bootstrap.cc:492] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:07.810586 9107 ts_tablet_manager.cc:1367] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.005s
I20240501 14:01:07.811565 9148 ts_tablet_manager.cc:902] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:07.811477 9107 raft_consensus.cc:348] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:07.811736 9107 raft_consensus.cc:729] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:01:07.811951 9107 consensus_queue.cc:260] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:07.814692 2006 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 6 to 7, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:07.816576 4899 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:07.821542 9113 tablet_copy_client.cc:783] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.821779 9111 tablet_copy_client.cc:783] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.822008 9111 tablet_copy_client.cc:647] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.822121 9113 tablet_copy_client.cc:647] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.822414 9066 raft_consensus.cc:348] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:07.823113 9066 raft_consensus.cc:729] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:01:07.823303 9066 consensus_queue.cc:260] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:07.823836 9066 ts_tablet_manager.cc:1398] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240501 14:01:07.827817 5054 tablet_copy_service.cc:342] P 6f81f803ec7746ccb502c59b7cbf8eb2: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-ea342bfcde3b4daa8dcc39126fd42cdc received from {username='slave'} at 127.28.17.3:48677
I20240501 14:01:07.827984 5054 tablet_copy_service.cc:433] P 6f81f803ec7746ccb502c59b7cbf8eb2: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-ea342bfcde3b4daa8dcc39126fd42cdc on tablet ea342bfcde3b4daa8dcc39126fd42cdc with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:07.831907 9111 tablet_copy_client.cc:523] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.834128 9113 tablet_copy_client.cc:523] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.836611 2408 raft_consensus.cc:1203] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:07.838344 9107 ts_tablet_manager.cc:1398] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.028s user 0.001s sys 0.000s
I20240501 14:01:07.848296 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-e8e883d12de74534a02ca9ccaada086f received from {username='slave'} at 127.28.17.8:50045
I20240501 14:01:07.848556 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-e8e883d12de74534a02ca9ccaada086f on tablet e8e883d12de74534a02ca9ccaada086f with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:01:07.853628 9113 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:07.859081 9111 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:07.863886 9116 tablet_copy_client.cc:308] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:01:07.876204 9072 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.876616 9072 tablet_bootstrap.cc:492] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:07.876873 9072 ts_tablet_manager.cc:1367] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.089s user 0.006s sys 0.016s
I20240501 14:01:07.877657 9072 raft_consensus.cc:348] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:07.877876 9072 raft_consensus.cc:729] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:07.878041 9072 consensus_queue.cc:260] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:07.883656 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 878eb076eee94aa8988b3588d5395cba from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:44257)
I20240501 14:01:07.883786 9072 ts_tablet_manager.cc:1398] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20240501 14:01:07.883809 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 878eb076eee94aa8988b3588d5395cba from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:44257: session id = a64f34785a7b44fa8070b0452a4e0ddd-878eb076eee94aa8988b3588d5395cba
I20240501 14:01:07.884629 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-44a4e92b144f47b5800506f4bc3eadb6 received from {username='slave'} at 127.28.17.2:40205
I20240501 14:01:07.884792 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-44a4e92b144f47b5800506f4bc3eadb6 on tablet 44a4e92b144f47b5800506f4bc3eadb6 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:07.886631 9119 tablet_copy_client.cc:783] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.889760 2298 tablet_copy_source_session.cc:205] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.888737 9119 tablet_copy_client.cc:647] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.891292 9116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878eb076eee94aa8988b3588d5395cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.896615 9119 tablet_copy_client.cc:523] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.905573 2145 raft_consensus.cc:1203] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:07.923679 9119 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
W20240501 14:01:07.924566 5228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.3:33537, user_credentials={real_user=slave}} blocked reactor thread for 52648.5us
I20240501 14:01:07.937505 5162 raft_consensus.cc:1203] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:07.940076 9116 tablet_copy_client.cc:783] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:07.940332 9116 tablet_copy_client.cc:647] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:07.944164 9116 tablet_copy_client.cc:523] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:07.934182 2146 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:07.950147 9116 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:07.959338 9148 tablet_copy_client.cc:308] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:07.961247 5162 consensus_queue.cc:237] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:07.964480 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 4aa75617adf64020af85f4cf2bd13917 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:50935)
I20240501 14:01:07.964674 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 4aa75617adf64020af85f4cf2bd13917 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:50935: session id = f8c60f2bbac541dcba4424ab726f9ad9-4aa75617adf64020af85f4cf2bd13917
I20240501 14:01:07.969468 2558 tablet_copy_source_session.cc:205] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:07.970468 9113 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:07.971119 9113 tablet_bootstrap.cc:492] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:01:07.971436 9113 ts_tablet_manager.cc:1367] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.118s user 0.042s sys 0.013s
I20240501 14:01:07.971802 9148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa75617adf64020af85f4cf2bd13917. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:07.972368 9113 raft_consensus.cc:348] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:07.972666 9113 raft_consensus.cc:729] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:01:07.972872 9113 consensus_queue.cc:260] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:07.973464 9113 ts_tablet_manager.cc:1398] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:07.974401 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-65306dbef405443fb4c7188ccb762b83 received from {username='slave'} at 127.28.17.5:52743
I20240501 14:01:07.974534 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-65306dbef405443fb4c7188ccb762b83 on tablet 65306dbef405443fb4c7188ccb762b83 with peer f8c60f2bbac541dcba4424ab726f9ad9
W20240501 14:01:07.979897 5097 consensus_peers.cc:485] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9a9e1d4df0a48f59b03349380cb6608. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:07.980906 4899 raft_consensus.cc:1261] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.981491 8988 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.988327 5034 raft_consensus.cc:1261] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:07.989245 8988 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:07.994242 8658 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:07.999219 4900 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:08.008611 2001 catalog_manager.cc:5484] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:08.033419 9093 ts_tablet_manager.cc:902] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:08.041244 5032 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:08.041733 9148 tablet_copy_client.cc:783] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:08.042347 9148 tablet_copy_client.cc:647] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:08.052882 9116 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.058548 9116 tablet_bootstrap.cc:492] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:01:08.060174 9116 ts_tablet_manager.cc:1367] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.110s user 0.022s sys 0.009s
I20240501 14:01:08.061170 9148 tablet_copy_client.cc:523] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:08.064266 9116 raft_consensus.cc:348] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:08.065601 9116 raft_consensus.cc:729] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:01:08.066128 9116 consensus_queue.cc:260] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:08.067090 9148 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:08.069353 9116 ts_tablet_manager.cc:1398] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:08.071352 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-878eb076eee94aa8988b3588d5395cba received from {username='slave'} at 127.28.17.3:44257
I20240501 14:01:08.071470 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-878eb076eee94aa8988b3588d5395cba on tablet 878eb076eee94aa8988b3588d5395cba with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:08.090827 9093 tablet_copy_client.cc:308] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:01:08.092058 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet a9d4e5590d124918ac4814a86b006b69 from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:39663)
I20240501 14:01:08.092269 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet a9d4e5590d124918ac4814a86b006b69 from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:39663: session id = 745ac12076e84effa5bde1ecbd6a32ec-a9d4e5590d124918ac4814a86b006b69
I20240501 14:01:08.096391 4921 tablet_copy_source_session.cc:205] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:08.098665 9093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4e5590d124918ac4814a86b006b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.109028 9093 tablet_copy_client.cc:783] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:08.112363 5293 raft_consensus.cc:1203] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:08.119199 9093 tablet_copy_client.cc:647] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:08.133991 9093 tablet_copy_client.cc:523] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:08.137840 9093 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:08.139783 4899 consensus_queue.cc:237] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:08.140156 2538 raft_consensus.cc:1203] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:08.141872 9148 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.147048 9148 tablet_bootstrap.cc:492] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:01:08.148624 9148 ts_tablet_manager.cc:1367] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.000s
I20240501 14:01:08.152456 9119 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.153100 9148 raft_consensus.cc:348] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:08.149641 9111 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.153944 9111 tablet_bootstrap.cc:492] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:08.154434 9111 ts_tablet_manager.cc:1367] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.295s user 0.054s sys 0.003s
I20240501 14:01:08.154630 9148 raft_consensus.cc:729] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:01:08.154947 9148 consensus_queue.cc:260] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:08.156677 9148 ts_tablet_manager.cc:1398] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:01:08.156026 9111 raft_consensus.cc:348] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:08.157461 9111 raft_consensus.cc:729] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:01:08.158187 9119 tablet_bootstrap.cc:492] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:01:08.157739 9111 consensus_queue.cc:260] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:08.160104 9111 ts_tablet_manager.cc:1398] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:01:08.161159 9119 ts_tablet_manager.cc:1367] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.235s user 0.053s sys 0.000s
I20240501 14:01:08.161319 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-6c74bacbf4464bab9c403e2153953474 received from {username='slave'} at 127.28.17.8:53689
I20240501 14:01:08.161439 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-6c74bacbf4464bab9c403e2153953474 on tablet 6c74bacbf4464bab9c403e2153953474 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:01:08.162889 9119 raft_consensus.cc:348] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:08.164058 9119 raft_consensus.cc:729] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:01:08.164340 9093 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.164791 9093 tablet_bootstrap.cc:492] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:08.165107 9093 ts_tablet_manager.cc:1367] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.000s
I20240501 14:01:08.164326 9119 consensus_queue.cc:260] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:08.165933 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-4aa75617adf64020af85f4cf2bd13917 received from {username='slave'} at 127.28.17.5:50935
I20240501 14:01:08.166047 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-4aa75617adf64020af85f4cf2bd13917 on tablet 4aa75617adf64020af85f4cf2bd13917 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:08.165978 9093 raft_consensus.cc:348] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:08.166410 9093 raft_consensus.cc:729] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:08.166653 9093 consensus_queue.cc:260] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:08.167714 9093 ts_tablet_manager.cc:1398] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:08.168860 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-a9d4e5590d124918ac4814a86b006b69 received from {username='slave'} at 127.28.17.4:39663
I20240501 14:01:08.169021 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-a9d4e5590d124918ac4814a86b006b69 on tablet a9d4e5590d124918ac4814a86b006b69 with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:08.168381 9119 ts_tablet_manager.cc:1398] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:08.170157 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-f1e048df08cf4f7eaacb78d6592d79b1 received from {username='slave'} at 127.28.17.6:41439
I20240501 14:01:08.170277 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-f1e048df08cf4f7eaacb78d6592d79b1 on tablet f1e048df08cf4f7eaacb78d6592d79b1 with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:08.174634 9149 raft_consensus.cc:1050] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:08.175521 9149 consensus_queue.cc:237] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:08.180270 4900 raft_consensus.cc:1203] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:08.185443 2277 raft_consensus.cc:1261] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.185658 2538 raft_consensus.cc:1261] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.186316 9156 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.186910 9156 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:01:08.188177 4834 consensus_peers.cc:485] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86abd8c2e86044f59c8074aa9a77db0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:08.188333 9183 ts_tablet_manager.cc:902] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:08.188611 5031 raft_consensus.cc:1261] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:08.189100 2408 raft_consensus.cc:1261] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.189682 9157 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.191020 8829 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:08.203132 9157 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:08.209877 2406 raft_consensus.cc:1203] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:08.209868 2537 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:08.213346 2001 catalog_manager.cc:5484] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:08.213541 2406 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:08.222489 2277 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:08.224519 5293 consensus_queue.cc:237] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:08.230082 2406 raft_consensus.cc:1261] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.232620 9149 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.238309 5293 raft_consensus.cc:1203] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:08.247378 2274 raft_consensus.cc:1261] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.248260 9149 consensus_queue.cc:1035] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 14:01:08.251533 4900 raft_consensus.cc:1203] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:08.252413 5032 raft_consensus.cc:1203] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:08.252242 9157 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:08.255507 9192 ts_tablet_manager.cc:902] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:08.258646 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3bc3084368de4add8809f10451d96894 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:08.266327 9183 tablet_copy_client.cc:308] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:08.263375 2005 catalog_manager.cc:5484] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:08.269563 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet b9ce2e195d834f6b986ac7273e7c6b78 from peer d1a414d381f44829a2510cb76d481887 ({username='slave'} at 127.28.17.1:53971)
I20240501 14:01:08.269728 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet b9ce2e195d834f6b986ac7273e7c6b78 from peer d1a414d381f44829a2510cb76d481887 at {username='slave'} at 127.28.17.1:53971: session id = d1a414d381f44829a2510cb76d481887-b9ce2e195d834f6b986ac7273e7c6b78
I20240501 14:01:08.270136 2406 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:08.272882 2558 tablet_copy_source_session.cc:205] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:08.274837 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc3084368de4add8809f10451d96894 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44554
I20240501 14:01:08.276870 2277 raft_consensus.cc:2932] T 3bc3084368de4add8809f10451d96894 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:08.282975 9183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ce2e195d834f6b986ac7273e7c6b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.289541 8954 raft_consensus.cc:1050] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER d1a414d381f44829a2510cb76d481887 to VOTER
I20240501 14:01:08.290840 9183 tablet_copy_client.cc:783] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:08.291095 9183 tablet_copy_client.cc:647] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:08.292273 5293 raft_consensus.cc:1261] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:08.293041 8871 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:08.290422 8954 consensus_queue.cc:237] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:08.296262 2408 raft_consensus.cc:1203] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:08.298763 9183 tablet_copy_client.cc:523] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:08.303874 9183 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:08.307629 2145 raft_consensus.cc:1261] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.308285 5296 raft_consensus.cc:1261] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.309288 5032 raft_consensus.cc:1261] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.309939 8722 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.310601 8954 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.311115 8954 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.322634 8871 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:08.328526 5292 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:08.334071 9198 tablet_replica.cc:330] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:01:08.334604 9198 raft_consensus.cc:2229] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:08.335147 9198 raft_consensus.cc:2258] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:08.336046 9198 ts_tablet_manager.cc:1875] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:08.336508 2005 catalog_manager.cc:5484] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:08.341635 5031 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:08.355702 9154 raft_consensus.cc:1050] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:01:08.356546 9154 consensus_queue.cc:237] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:08.361191 2277 raft_consensus.cc:1203] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:01:08.366447 9157 raft_consensus.cc:1050] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:01:08.367412 9157 consensus_queue.cc:237] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:08.374702 8633 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.380347 5296 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.381223 9183 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.381570 9183 tablet_bootstrap.cc:492] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: Bootstrap complete.
I20240501 14:01:08.381819 9183 ts_tablet_manager.cc:1367] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.078s user 0.018s sys 0.003s
I20240501 14:01:08.382622 9183 raft_consensus.cc:348] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:08.382864 9183 raft_consensus.cc:729] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: LEARNER
I20240501 14:01:08.383042 9183 consensus_queue.cc:260] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:08.386960 9183 ts_tablet_manager.cc:1398] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:01:08.391587 2000 catalog_manager.cc:5484] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.393898 2147 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.397940 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session d1a414d381f44829a2510cb76d481887-b9ce2e195d834f6b986ac7273e7c6b78 received from {username='slave'} at 127.28.17.1:53971
I20240501 14:01:08.398098 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session d1a414d381f44829a2510cb76d481887-b9ce2e195d834f6b986ac7273e7c6b78 on tablet b9ce2e195d834f6b986ac7273e7c6b78 with peer d1a414d381f44829a2510cb76d481887
I20240501 14:01:08.407336 9198 ts_tablet_manager.cc:1888] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:08.408571 5031 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.409550 9198 log.cc:1196] T 3bc3084368de4add8809f10451d96894 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/3bc3084368de4add8809f10451d96894
I20240501 14:01:08.410899 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 3bc3084368de4add8809f10451d96894 (table rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]) successfully deleted
I20240501 14:01:08.420035 2537 consensus_queue.cc:237] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:08.423305 5292 raft_consensus.cc:1261] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.424075 8954 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.425458 2147 raft_consensus.cc:1261] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.426019 8954 consensus_queue.cc:1035] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.429853 8954 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.433166 5296 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.433367 2147 raft_consensus.cc:2932] T f6e71ec5dcbe4a08a0a7da5944dfe77d P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.438376 2408 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.438984 8976 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.443977 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:08.452940 2005 catalog_manager.cc:5484] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:08.457602 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e71ec5dcbe4a08a0a7da5944dfe77d with delete_type TABLET_DATA_TOMBSTONED (TS 6f81f803ec7746ccb502c59b7cbf8eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58280
I20240501 14:01:08.460104 9216 ts_tablet_manager.cc:902] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:08.471087 2277 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.471678 4899 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.472121 9154 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.472996 8976 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.482954 4900 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.484292 9149 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.485132 5164 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.486321 2537 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.489840 9149 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.490450 9149 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.493558 8976 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.499789 2277 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.500128 4900 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.510313 2005 catalog_manager.cc:5484] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.520560 8658 raft_consensus.cc:1050] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:08.521457 8658 consensus_queue.cc:237] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:08.525357 2406 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.532903 9157 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:08.537427 4899 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:08.540119 5162 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:08.544381 2537 raft_consensus.cc:1261] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.544852 5296 raft_consensus.cc:1261] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.545085 8988 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.545050 2005 catalog_manager.cc:5484] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:08.545989 9174 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.545940 2537 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:08.551906 2274 raft_consensus.cc:1261] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.555873 8988 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.557550 5031 consensus_queue.cc:237] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:08.560425 9217 tablet_replica.cc:330] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:01:08.560781 9217 raft_consensus.cc:2229] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:08.561210 9217 raft_consensus.cc:2258] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:08.561748 9192 tablet_copy_client.cc:308] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:01:08.567438 2537 raft_consensus.cc:1203] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:08.571106 2406 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.572872 9049 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.575500 2277 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.576205 9049 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.576689 9154 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.578217 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet fd3f8661a13944098a18d3eea7ddeb00 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:46539)
I20240501 14:01:08.578369 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet fd3f8661a13944098a18d3eea7ddeb00 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:46539: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-fd3f8661a13944098a18d3eea7ddeb00
I20240501 14:01:08.579005 2277 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.580412 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7b11568c2d7420fa06f903e52723583 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:08.581113 2428 tablet_copy_source_session.cc:205] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:08.585186 9174 raft_consensus.cc:1050] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:08.585318 9212 raft_consensus.cc:1050] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:01:08.586030 9174 consensus_queue.cc:237] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:08.586097 9212 consensus_queue.cc:237] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:08.586256 2408 raft_consensus.cc:2932] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.589242 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f8661a13944098a18d3eea7ddeb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.590514 2001 catalog_manager.cc:5484] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.595309 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b11568c2d7420fa06f903e52723583 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:08.596437 9192 tablet_copy_client.cc:783] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:08.596704 9192 tablet_copy_client.cc:647] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:08.606472 5296 raft_consensus.cc:1261] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.607120 9174 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.607726 5031 raft_consensus.cc:1261] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.608336 8658 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.610004 2537 raft_consensus.cc:1261] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.611366 9174 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240501 14:01:08.614624 9192 tablet_copy_client.cc:523] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:08.623819 8658 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.629207 5292 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.629528 2533 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.636500 2001 catalog_manager.cc:5484] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.646255 9192 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:08.647696 8988 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.651502 5164 consensus_queue.cc:237] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:08.654742 5293 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.656441 2537 raft_consensus.cc:1261] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.656682 5292 raft_consensus.cc:1261] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.662783 2005 catalog_manager.cc:5484] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:08.663249 8988 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.664088 8988 consensus_queue.cc:1035] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.665683 9217 ts_tablet_manager.cc:1875] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:08.661432 2538 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.669693 9032 raft_consensus.cc:1050] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:01:08.670672 9032 consensus_queue.cc:237] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:08.679730 2277 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:08.678256 5164 consensus_queue.cc:237] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:08.680519 8835 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:08.680858 5032 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.688704 2147 raft_consensus.cc:1203] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:08.694818 5031 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:08.695989 9032 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:08.700698 5296 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:08.701269 8835 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:08.702482 9217 ts_tablet_manager.cc:1888] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:08.702648 9217 log.cc:1196] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/f6e71ec5dcbe4a08a0a7da5944dfe77d
I20240501 14:01:08.705108 9216 tablet_copy_client.cc:308] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:08.705154 8835 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.706247 4899 raft_consensus.cc:1261] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.706913 8722 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.710947 2274 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.715763 2001 catalog_manager.cc:5484] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 7 to 8, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:08.712987 5293 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.720296 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet e9a9e1d4df0a48f59b03349380cb6608 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:40205)
I20240501 14:01:08.720485 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet e9a9e1d4df0a48f59b03349380cb6608 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:40205: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-e9a9e1d4df0a48f59b03349380cb6608
I20240501 14:01:08.722647 5296 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.727602 2147 raft_consensus.cc:1261] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.728554 9212 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.729558 8868 raft_consensus.cc:1050] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:01:08.730446 8868 consensus_queue.cc:237] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:08.740967 5184 tablet_copy_source_session.cc:205] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:08.742368 5032 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.737130 9174 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.749785 9235 tablet_replica.cc:330] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:08.750108 9235 raft_consensus.cc:2229] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:08.750516 9235 raft_consensus.cc:2258] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:08.758020 2001 catalog_manager.cc:5484] T 8cdc95819b944c5590202e86cc955d91 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.763908 2406 consensus_queue.cc:237] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:08.764425 9235 ts_tablet_manager.cc:1875] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:08.768036 5296 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:08.768581 5031 raft_consensus.cc:1261] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:08.768604 9194 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:08.769171 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cdc95819b944c5590202e86cc955d91 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35676
I20240501 14:01:08.770548 9194 consensus_queue.cc:1035] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:08.772276 9255 tablet_replica.cc:330] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:01:08.773304 9253 ts_tablet_manager.cc:902] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:08.773180 9194 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.774187 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet f6e71ec5dcbe4a08a0a7da5944dfe77d (table rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]) successfully deleted
I20240501 14:01:08.772595 9255 raft_consensus.cc:2229] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:08.774878 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cdc95819b944c5590202e86cc955d91 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:08.775238 9255 pending_rounds.cc:62] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Trying to abort 1 pending ops.
I20240501 14:01:08.775473 9255 pending_rounds.cc:68] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022887192661479424 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8cdc95819b944c5590202e86cc955d91" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } } }
I20240501 14:01:08.776360 9255 raft_consensus.cc:2894] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:01:08.776680 9255 raft_consensus.cc:2866] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:01:08.776921 9255 raft_consensus.cc:2258] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:08.778100 9255 ts_tablet_manager.cc:1875] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:08.778710 5034 raft_consensus.cc:1261] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.778986 9216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a9e1d4df0a48f59b03349380cb6608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:08.780174 5292 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.782073 8633 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.783421 5034 raft_consensus.cc:2932] T f1e048df08cf4f7eaacb78d6592d79b1 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.788810 2003 catalog_manager.cc:5484] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:08.791178 9224 raft_consensus.cc:1050] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:01:08.791990 9224 consensus_queue.cc:237] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:08.794675 2408 raft_consensus.cc:1261] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240501 14:01:08.795007 5097 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.3:33537, user_credentials={real_user=slave}} blocked reactor thread for 132812us
I20240501 14:01:08.796037 2537 raft_consensus.cc:1261] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.796051 5294 raft_consensus.cc:1261] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.797241 8658 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.800509 8988 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.801103 8663 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.802048 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e048df08cf4f7eaacb78d6592d79b1 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35676
I20240501 14:01:08.803057 5294 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.804731 9255 ts_tablet_manager.cc:1888] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:08.804850 9255 log.cc:1196] T 8cdc95819b944c5590202e86cc955d91 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/8cdc95819b944c5590202e86cc955d91
I20240501 14:01:08.805565 9255 tablet_replica.cc:330] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:01:08.805756 9255 raft_consensus.cc:2229] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:08.806159 9255 raft_consensus.cc:2258] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:08.807461 9255 ts_tablet_manager.cc:1875] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:08.809437 8658 consensus_queue.cc:1035] T e8e883d12de74534a02ca9ccaada086f P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.810189 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet 8cdc95819b944c5590202e86cc955d91 (table rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]) successfully deleted
I20240501 14:01:08.811064 5293 raft_consensus.cc:1261] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.811707 8663 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.812538 2000 catalog_manager.cc:5484] T e8e883d12de74534a02ca9ccaada086f P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.821547 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e883d12de74534a02ca9ccaada086f with delete_type TABLET_DATA_TOMBSTONED (TS 6f81f803ec7746ccb502c59b7cbf8eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58280
I20240501 14:01:08.821947 9217 tablet_replica.cc:330] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:01:08.822172 9217 raft_consensus.cc:2229] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:08.822485 9217 raft_consensus.cc:2258] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:08.823325 9217 ts_tablet_manager.cc:1875] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:08.826872 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1e048df08cf4f7eaacb78d6592d79b1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:08.830056 8663 raft_consensus.cc:1050] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:01:08.831061 8663 consensus_queue.cc:237] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:08.838258 9217 ts_tablet_manager.cc:1888] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:08.838797 9217 log.cc:1196] T e8e883d12de74534a02ca9ccaada086f P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/e8e883d12de74534a02ca9ccaada086f
I20240501 14:01:08.850546 2408 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.850555 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet e8e883d12de74534a02ca9ccaada086f (table rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]) successfully deleted
I20240501 14:01:08.851418 9049 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.852870 4899 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.853588 9224 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.854236 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e8e883d12de74534a02ca9ccaada086f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:08.860411 2538 raft_consensus.cc:2932] T e8e883d12de74534a02ca9ccaada086f P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.870559 2537 raft_consensus.cc:2932] T 8cdc95819b944c5590202e86cc955d91 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:08.875437 9032 raft_consensus.cc:1050] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:08.876312 9032 consensus_queue.cc:237] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:08.882702 5162 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.883706 9216 tablet_copy_client.cc:783] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:08.884066 9216 tablet_copy_client.cc:647] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:08.889988 5034 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:08.892565 5162 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:08.893488 5162 raft_consensus.cc:1261] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:08.899727 2533 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:08.900409 9206 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:08.902673 9049 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.902877 9255 ts_tablet_manager.cc:1888] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:08.902973 5032 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:08.903155 9255 log.cc:1196] T f1e048df08cf4f7eaacb78d6592d79b1 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1e048df08cf4f7eaacb78d6592d79b1
I20240501 14:01:08.903301 9049 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.905007 8722 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:08.909783 4900 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:08.911177 9194 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:08.912210 5292 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:08.915037 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet f1e048df08cf4f7eaacb78d6592d79b1 (table rebalance_test_table_9 [id=761b589d15b74dbaa0b68eb1e7848552]) successfully deleted
I20240501 14:01:08.915796 2003 catalog_manager.cc:5484] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:08.916466 9206 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:08.916640 9265 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:08.917197 9216 tablet_copy_client.cc:523] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:08.921201 9216 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
W20240501 14:01:08.922935 4967 consensus_peers.cc:485] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf051cecef424db5accfce66623346b5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:08.924839 5032 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:08.925302 4901 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:08.930253 8976 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:08.932066 2533 consensus_queue.cc:237] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:08.936805 4898 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.940372 5031 raft_consensus.cc:1261] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.940940 9212 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.942409 4900 raft_consensus.cc:1261] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:08.942852 2408 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:08.944087 9049 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:08.944820 2405 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.947554 2003 catalog_manager.cc:5484] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.948971 9212 consensus_queue.cc:1035] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:08.951932 8722 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:08.957005 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4aa75617adf64020af85f4cf2bd13917 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:08.959645 4900 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:08.962571 5032 raft_consensus.cc:2932] T 4aa75617adf64020af85f4cf2bd13917 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:08.964175 9235 ts_tablet_manager.cc:1888] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:08.965077 9268 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.967298 9235 log.cc:1196] T a7b11568c2d7420fa06f903e52723583 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/a7b11568c2d7420fa06f903e52723583
I20240501 14:01:08.967489 2005 catalog_manager.cc:5484] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:08.974920 5161 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.893571 9264 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:08.972741 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet a7b11568c2d7420fa06f903e52723583 (table rebalancer_extra_table_14 [id=b9864581dc3c485fbb864295c3c019f1]) successfully deleted
I20240501 14:01:08.988507 2003 catalog_manager.cc:5484] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:08.998831 9032 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:08.999027 8976 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:09.002954 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa75617adf64020af85f4cf2bd13917 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48706
I20240501 14:01:09.003551 5293 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.006390 2537 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.012091 2407 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:09.018751 5031 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.025698 2006 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:09.026825 2147 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:09.026911 2004 catalog_manager.cc:5484] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.027551 9264 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:09.035652 9216 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:09.036132 9216 tablet_bootstrap.cc:492] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:09.036502 9216 ts_tablet_manager.cc:1367] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.003s
I20240501 14:01:09.024783 2405 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.038031 5034 consensus_queue.cc:237] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:09.041366 2408 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.041963 9277 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.051246 8663 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.055052 9216 raft_consensus.cc:348] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:09.055301 9216 raft_consensus.cc:729] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:09.055502 9216 consensus_queue.cc:260] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:09.056118 9216 ts_tablet_manager.cc:1398] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:01:09.056671 4897 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.058068 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-e9a9e1d4df0a48f59b03349380cb6608 received from {username='slave'} at 127.28.17.2:40205
I20240501 14:01:09.058234 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-e9a9e1d4df0a48f59b03349380cb6608 on tablet e9a9e1d4df0a48f59b03349380cb6608 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:09.059128 9277 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.059990 8871 raft_consensus.cc:1050] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:09.060899 8871 consensus_queue.cc:237] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:09.061079 4901 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:09.061939 2147 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.067570 9224 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.072096 4898 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.076175 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ea342bfcde3b4daa8dcc39126fd42cdc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.081279 2407 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.081557 2004 catalog_manager.cc:5484] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.083238 2537 raft_consensus.cc:1261] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.083385 5293 raft_consensus.cc:1261] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.083876 2146 raft_consensus.cc:1261] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.084554 8829 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.085222 8871 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.085672 9208 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.089040 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet ea342bfcde3b4daa8dcc39126fd42cdc with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:09.107072 5163 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.114617 9285 tablet_replica.cc:330] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:01:09.116051 4898 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:09.120747 9264 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:09.126834 9253 tablet_copy_client.cc:308] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:01:09.127625 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet 86abd8c2e86044f59c8074aa9a77db0d from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:39663)
I20240501 14:01:09.127863 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet 86abd8c2e86044f59c8074aa9a77db0d from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:39663: session id = 745ac12076e84effa5bde1ecbd6a32ec-86abd8c2e86044f59c8074aa9a77db0d
I20240501 14:01:09.128644 4899 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.132946 9208 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.135521 9285 raft_consensus.cc:2229] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.135759 9285 pending_rounds.cc:62] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Trying to abort 1 pending ops.
I20240501 14:01:09.135802 9285 pending_rounds.cc:68] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022887192926400512 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4aa75617adf64020af85f4cf2bd13917" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } } }
I20240501 14:01:09.136183 9285 raft_consensus.cc:2894] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:01:09.136289 9285 raft_consensus.cc:2866] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:01:09.136375 9285 raft_consensus.cc:2258] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.137279 4921 tablet_copy_source_session.cc:205] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:09.139053 9253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86abd8c2e86044f59c8074aa9a77db0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.138895 5293 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.139339 2001 catalog_manager.cc:5484] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 7 to 8, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.145035 2146 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.146348 9285 ts_tablet_manager.cc:1875] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.155043 2537 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.159628 2004 catalog_manager.cc:5484] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.169900 9253 tablet_copy_client.cc:783] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:09.171710 9253 tablet_copy_client.cc:647] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:09.174098 4901 consensus_queue.cc:237] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:09.175392 9253 tablet_copy_client.cc:523] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:09.176057 5162 raft_consensus.cc:2932] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.177256 2146 raft_consensus.cc:1261] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.177800 8829 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.181275 8829 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.185559 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9d4e5590d124918ac4814a86b006b69 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.186890 2147 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.189832 9253 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:09.191449 2538 raft_consensus.cc:1261] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.193421 9292 tablet_replica.cc:330] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:09.193673 9292 raft_consensus.cc:2229] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.193940 9292 raft_consensus.cc:2258] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.195202 2004 catalog_manager.cc:5484] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.199800 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d4e5590d124918ac4814a86b006b69 with delete_type TABLET_DATA_TOMBSTONED (TS 5a4b9a96e9734baa8a03d6056bfa934c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36154
I20240501 14:01:09.201159 9208 consensus_queue.cc:1035] T a9d4e5590d124918ac4814a86b006b69 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.203886 2537 raft_consensus.cc:2932] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.212183 9297 tablet_replica.cc:330] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:01:09.212497 9297 raft_consensus.cc:2229] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.212844 9297 raft_consensus.cc:2258] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.213768 9297 ts_tablet_manager.cc:1875] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.216688 9285 ts_tablet_manager.cc:1888] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:09.219331 9292 ts_tablet_manager.cc:1875] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.220446 9297 ts_tablet_manager.cc:1888] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:09.220566 9297 log.cc:1196] T a9d4e5590d124918ac4814a86b006b69 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/a9d4e5590d124918ac4814a86b006b69
I20240501 14:01:09.221395 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet a9d4e5590d124918ac4814a86b006b69 (table rebalancer_extra_table_26 [id=d29775b4e5724c54b5927610b54e6144]) successfully deleted
I20240501 14:01:09.228367 9292 ts_tablet_manager.cc:1888] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:09.228646 9292 log.cc:1196] T ea342bfcde3b4daa8dcc39126fd42cdc P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/ea342bfcde3b4daa8dcc39126fd42cdc
I20240501 14:01:09.229370 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet ea342bfcde3b4daa8dcc39126fd42cdc (table rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]) successfully deleted
I20240501 14:01:09.236896 9285 log.cc:1196] T 4aa75617adf64020af85f4cf2bd13917 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/4aa75617adf64020af85f4cf2bd13917
I20240501 14:01:09.236932 8722 raft_consensus.cc:1050] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER d1a414d381f44829a2510cb76d481887 to VOTER
I20240501 14:01:09.238152 2407 consensus_queue.cc:237] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:09.237756 8722 consensus_queue.cc:237] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:09.241364 5034 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:09.242197 8835 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:09.246534 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 4aa75617adf64020af85f4cf2bd13917 (table rebalancer_extra_table_44 [id=c6e92798845b46c8bc8476bb4c731ab3]) successfully deleted
I20240501 14:01:09.249590 8835 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.256325 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c74bacbf4464bab9c403e2153953474 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.265436 9253 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:09.265988 9253 tablet_bootstrap.cc:492] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:09.266347 9253 ts_tablet_manager.cc:1367] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.007s
I20240501 14:01:09.268708 5293 raft_consensus.cc:1261] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:09.270083 9253 raft_consensus.cc:348] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:09.270555 9253 raft_consensus.cc:729] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:09.270814 9253 consensus_queue.cc:260] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:09.283532 9252 raft_consensus.cc:1050] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:09.284451 9252 consensus_queue.cc:237] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:09.287833 9253 ts_tablet_manager.cc:1398] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240501 14:01:09.287778 8835 consensus_queue.cc:1035] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:09.288982 5032 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.290120 5293 raft_consensus.cc:2932] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.294919 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-86abd8c2e86044f59c8074aa9a77db0d received from {username='slave'} at 127.28.17.4:39663
I20240501 14:01:09.295089 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-86abd8c2e86044f59c8074aa9a77db0d on tablet 86abd8c2e86044f59c8074aa9a77db0d with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:09.301388 5293 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.310663 2000 catalog_manager.cc:5484] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 8 to 9, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:09.314057 5034 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:09.314975 9252 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:09.322113 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c74bacbf4464bab9c403e2153953474 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44554
I20240501 14:01:09.323329 9306 tablet_replica.cc:330] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:01:09.323632 9306 raft_consensus.cc:2229] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.323896 9306 raft_consensus.cc:2258] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.325055 9306 ts_tablet_manager.cc:1875] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.326887 9192 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:09.327867 9192 tablet_bootstrap.cc:492] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:01:09.328213 9192 ts_tablet_manager.cc:1367] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.682s user 0.050s sys 0.016s
I20240501 14:01:09.328886 5293 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:09.329221 9192 raft_consensus.cc:348] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:09.329505 9192 raft_consensus.cc:729] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:01:09.329536 9252 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:09.329716 9192 consensus_queue.cc:260] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:09.332321 2274 raft_consensus.cc:1203] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:09.332541 9192 ts_tablet_manager.cc:1398] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:01:09.333194 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-fd3f8661a13944098a18d3eea7ddeb00 received from {username='slave'} at 127.28.17.7:46539
I20240501 14:01:09.333319 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-fd3f8661a13944098a18d3eea7ddeb00 on tablet fd3f8661a13944098a18d3eea7ddeb00 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:09.333470 2274 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:09.336130 2147 raft_consensus.cc:1261] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.337098 9306 ts_tablet_manager.cc:1888] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:09.337222 9306 log.cc:1196] T 6c74bacbf4464bab9c403e2153953474 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c74bacbf4464bab9c403e2153953474
I20240501 14:01:09.337281 2405 raft_consensus.cc:1261] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.338120 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet 6c74bacbf4464bab9c403e2153953474 (table rebalance_test_table_1 [id=b00206616c0445939acfde48c1f27555]) successfully deleted
I20240501 14:01:09.338279 9212 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.338883 9212 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.338970 4900 raft_consensus.cc:1261] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.340142 8954 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.351583 2537 raft_consensus.cc:1203] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:09.391036 2274 consensus_queue.cc:237] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:09.393875 2407 raft_consensus.cc:1261] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.394848 9271 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.400276 9241 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:09.403802 8663 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.407961 2407 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.411839 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 878eb076eee94aa8988b3588d5395cba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.427980 5293 raft_consensus.cc:1261] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.428740 9271 consensus_queue.cc:1035] T 878eb076eee94aa8988b3588d5395cba P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.430341 5294 raft_consensus.cc:2932] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.435079 9252 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.435896 9212 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:09.444625 5293 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.446769 1998 catalog_manager.cc:5484] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 5 to 6, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:09.447479 2146 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:09.448468 2405 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:09.421962 2006 catalog_manager.cc:5484] T 878eb076eee94aa8988b3588d5395cba P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:09.450852 5034 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.455482 4900 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:09.459630 2001 catalog_manager.cc:5484] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:09.468391 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 878eb076eee94aa8988b3588d5395cba with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:09.468778 9292 tablet_replica.cc:330] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:09.468979 9292 raft_consensus.cc:2229] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.469256 9292 raft_consensus.cc:2258] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.470732 9292 ts_tablet_manager.cc:1875] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.488265 2274 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.499994 5162 consensus_queue.cc:237] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:09.501112 2537 consensus_queue.cc:237] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:09.502908 9292 ts_tablet_manager.cc:1888] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:09.503105 9292 log.cc:1196] T 878eb076eee94aa8988b3588d5395cba P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/878eb076eee94aa8988b3588d5395cba
I20240501 14:01:09.504107 2405 raft_consensus.cc:1261] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.504130 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 878eb076eee94aa8988b3588d5395cba (table rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]) successfully deleted
I20240501 14:01:09.504916 2146 raft_consensus.cc:1261] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.505596 5293 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:09.506575 9174 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:09.507079 8722 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.507608 8722 consensus_queue.cc:1035] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.509749 9174 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.514770 5293 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.518158 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44a4e92b144f47b5800506f4bc3eadb6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.518697 8633 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:09.521896 2006 catalog_manager.cc:5484] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 6 to 7, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:09.526099 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9ce2e195d834f6b986ac7273e7c6b78 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.525830 2405 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:09.530936 2005 catalog_manager.cc:5484] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:09.532047 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 44a4e92b144f47b5800506f4bc3eadb6 with delete_type TABLET_DATA_TOMBSTONED (TS 6f81f803ec7746ccb502c59b7cbf8eb2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58280
I20240501 14:01:09.538127 2274 raft_consensus.cc:1261] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:09.543687 9241 consensus_queue.cc:1035] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:09.548027 2274 raft_consensus.cc:2932] T 44a4e92b144f47b5800506f4bc3eadb6 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.554896 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ce2e195d834f6b986ac7273e7c6b78 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:09.564891 2146 raft_consensus.cc:2932] T b9ce2e195d834f6b986ac7273e7c6b78 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:09.572805 9325 tablet_replica.cc:330] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:09.584373 9323 tablet_replica.cc:330] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:01:09.584657 9324 ts_tablet_manager.cc:902] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: Initiating tablet copy from peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:09.584940 9323 raft_consensus.cc:2229] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.585850 9323 raft_consensus.cc:2258] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.587790 9325 raft_consensus.cc:2229] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.588146 9325 raft_consensus.cc:2258] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.588938 9323 ts_tablet_manager.cc:1875] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.589040 9325 ts_tablet_manager.cc:1875] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.603499 9325 ts_tablet_manager.cc:1888] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:09.603749 9325 log.cc:1196] T b9ce2e195d834f6b986ac7273e7c6b78 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/b9ce2e195d834f6b986ac7273e7c6b78
I20240501 14:01:09.604593 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet b9ce2e195d834f6b986ac7273e7c6b78 (table rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]) successfully deleted
I20240501 14:01:09.607574 9323 ts_tablet_manager.cc:1888] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:01:09.607863 9323 log.cc:1196] T 44a4e92b144f47b5800506f4bc3eadb6 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/44a4e92b144f47b5800506f4bc3eadb6
I20240501 14:01:09.609272 5162 raft_consensus.cc:1203] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:09.610813 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 44a4e92b144f47b5800506f4bc3eadb6 (table rebalancer_extra_table_2 [id=949b504259924baa83f0a25886f72a26]) successfully deleted
I20240501 14:01:09.612082 9324 tablet_copy_client.cc:308] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.6:46037
I20240501 14:01:09.612867 5054 tablet_copy_service.cc:140] P 6f81f803ec7746ccb502c59b7cbf8eb2: Received BeginTabletCopySession request for tablet bf051cecef424db5accfce66623346b5 from peer d1a414d381f44829a2510cb76d481887 ({username='slave'} at 127.28.17.1:41427)
I20240501 14:01:09.613044 5054 tablet_copy_service.cc:161] P 6f81f803ec7746ccb502c59b7cbf8eb2: Beginning new tablet copy session on tablet bf051cecef424db5accfce66623346b5 from peer d1a414d381f44829a2510cb76d481887 at {username='slave'} at 127.28.17.1:41427: session id = d1a414d381f44829a2510cb76d481887-bf051cecef424db5accfce66623346b5
I20240501 14:01:09.615785 5054 tablet_copy_source_session.cc:205] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:09.617997 9324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf051cecef424db5accfce66623346b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:09.626338 9324 tablet_copy_client.cc:783] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:09.626648 9324 tablet_copy_client.cc:647] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:09.628355 9324 tablet_copy_client.cc:523] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:09.632357 9324 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:09.633981 2274 consensus_queue.cc:237] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:09.636364 4900 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:09.637992 9271 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:09.639839 5162 raft_consensus.cc:1261] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:09.644518 9271 consensus_queue.cc:1035] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:09.651631 9271 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.656560 5162 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.660979 4900 raft_consensus.cc:2932] T 65306dbef405443fb4c7188ccb762b83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:09.681773 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 65306dbef405443fb4c7188ccb762b83 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.684073 2005 catalog_manager.cc:5484] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:09.695226 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 65306dbef405443fb4c7188ccb762b83 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48706
I20240501 14:01:09.695587 9285 tablet_replica.cc:330] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:01:09.695775 9285 raft_consensus.cc:2229] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.696027 9285 raft_consensus.cc:2258] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.696995 9285 ts_tablet_manager.cc:1875] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.704507 9285 ts_tablet_manager.cc:1888] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:09.704742 9285 log.cc:1196] T 65306dbef405443fb4c7188ccb762b83 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/65306dbef405443fb4c7188ccb762b83
I20240501 14:01:09.705578 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 65306dbef405443fb4c7188ccb762b83 (table rebalance_test_table_11 [id=be2c45ef0f17476d82da5823f508a3ba]) successfully deleted
I20240501 14:01:09.716595 9324 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:09.718356 9324 tablet_bootstrap.cc:492] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: Bootstrap complete.
I20240501 14:01:09.718644 9324 ts_tablet_manager.cc:1367] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.011s
I20240501 14:01:09.719336 9324 raft_consensus.cc:348] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:09.719573 9324 raft_consensus.cc:729] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: LEARNER
I20240501 14:01:09.719740 9324 consensus_queue.cc:260] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:09.725466 9324 ts_tablet_manager.cc:1398] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:09.735553 5054 tablet_copy_service.cc:342] P 6f81f803ec7746ccb502c59b7cbf8eb2: Request end of tablet copy session d1a414d381f44829a2510cb76d481887-bf051cecef424db5accfce66623346b5 received from {username='slave'} at 127.28.17.1:41427
I20240501 14:01:09.735708 5054 tablet_copy_service.cc:433] P 6f81f803ec7746ccb502c59b7cbf8eb2: ending tablet copy session d1a414d381f44829a2510cb76d481887-bf051cecef424db5accfce66623346b5 on tablet bf051cecef424db5accfce66623346b5 with peer d1a414d381f44829a2510cb76d481887
I20240501 14:01:09.742197 5293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c"
dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60284
I20240501 14:01:09.742609 5293 raft_consensus.cc:595] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:09.822873 9252 raft_consensus.cc:1050] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:09.823765 9252 consensus_queue.cc:237] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:09.836261 2274 raft_consensus.cc:1261] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.836342 5032 raft_consensus.cc:1261] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.836989 9241 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.837425 9174 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.839000 4901 raft_consensus.cc:1261] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.841284 9241 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.854108 9208 raft_consensus.cc:1050] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:09.855142 9208 consensus_queue.cc:237] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:09.866379 2537 raft_consensus.cc:1261] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.867163 8829 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.875890 9390 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.880755 4901 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.882468 2274 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.889081 5034 raft_consensus.cc:1261] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.889459 2000 catalog_manager.cc:5484] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.888895 5032 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.891196 9208 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.891408 5293 raft_consensus.cc:1261] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:09.893150 8829 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:09.900465 5160 consensus_queue.cc:237] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:09.907886 2274 raft_consensus.cc:1261] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.908828 9241 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.911336 4901 raft_consensus.cc:1261] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.912042 9241 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.913292 9252 consensus_queue.cc:1035] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.914971 4901 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.915664 2274 raft_consensus.cc:2932] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.921015 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9a9e1d4df0a48f59b03349380cb6608 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.925825 2005 catalog_manager.cc:5484] T e9a9e1d4df0a48f59b03349380cb6608 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:09.931078 8829 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.935391 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet e9a9e1d4df0a48f59b03349380cb6608 with delete_type TABLET_DATA_TOMBSTONED (TS 6f81f803ec7746ccb502c59b7cbf8eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58280
I20240501 14:01:09.935818 9323 tablet_replica.cc:330] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:01:09.935521 5294 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.936041 9323 raft_consensus.cc:2229] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.936306 9323 raft_consensus.cc:2258] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.939450 2537 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.945407 5032 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.951311 9323 ts_tablet_manager.cc:1875] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.954078 2005 catalog_manager.cc:5484] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.964210 9323 ts_tablet_manager.cc:1888] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:09.964458 9323 log.cc:1196] T e9a9e1d4df0a48f59b03349380cb6608 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/e9a9e1d4df0a48f59b03349380cb6608
I20240501 14:01:09.965871 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet e9a9e1d4df0a48f59b03349380cb6608 (table rebalancer_extra_table_22 [id=2fbbdc4802104277803b37a22211ab14]) successfully deleted
I20240501 14:01:09.969831 4901 consensus_queue.cc:237] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:09.972292 2537 raft_consensus.cc:1261] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.972965 5032 raft_consensus.cc:1261] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:09.973584 8829 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.974262 8829 consensus_queue.cc:1035] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:09.977411 8829 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.980595 5034 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.982287 2537 raft_consensus.cc:2932] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.984903 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86abd8c2e86044f59c8074aa9a77db0d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.989711 2000 catalog_manager.cc:5484] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:09.993227 9149 raft_consensus.cc:979] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: : Instructing follower f8c60f2bbac541dcba4424ab726f9ad9 to start an election
I20240501 14:01:09.993496 9157 raft_consensus.cc:1067] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Signalling peer f8c60f2bbac541dcba4424ab726f9ad9 to start an election
I20240501 14:01:09.994768 4901 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c"
dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
from {username='slave'} at 127.28.17.8:38093
I20240501 14:01:09.994985 4901 raft_consensus.cc:482] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:09.995112 4901 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:09.998351 4901 raft_consensus.cc:504] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:09.999768 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 86abd8c2e86044f59c8074aa9a77db0d with delete_type TABLET_DATA_TOMBSTONED (TS 5a4b9a96e9734baa8a03d6056bfa934c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36154
I20240501 14:01:09.999799 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:09.999975 2537 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:10.008198 2537 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:10.009192 4901 leader_election.cc:288] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:10.009469 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:10.009526 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:10.009711 5160 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:10.009721 5293 raft_consensus.cc:3032] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:10.009806 5293 raft_consensus.cc:729] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:01:10.010123 5293 consensus_queue.cc:260] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:10.010624 5293 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:10.012599 5160 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:10.015808 4834 leader_election.cc:302] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:10.016330 5293 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:10.016618 8829 raft_consensus.cc:2781] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:10.016767 8829 raft_consensus.cc:686] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:10.017042 8829 consensus_queue.cc:237] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:10.019567 9404 tablet_replica.cc:330] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:01:10.019829 9404 raft_consensus.cc:2229] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:10.020089 9404 raft_consensus.cc:2258] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:10.023429 9404 ts_tablet_manager.cc:1875] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:10.046662 2005 catalog_manager.cc:5484] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.057546 9404 ts_tablet_manager.cc:1888] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:10.057880 9404 log.cc:1196] T 86abd8c2e86044f59c8074aa9a77db0d P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/86abd8c2e86044f59c8074aa9a77db0d
I20240501 14:01:10.059196 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 86abd8c2e86044f59c8074aa9a77db0d (table rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]) successfully deleted
I20240501 14:01:10.079623 9032 raft_consensus.cc:1050] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:01:10.080487 9032 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:10.083257 5162 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:10.084092 2147 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:10.084906 4901 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:10.085021 9206 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:10.085722 9206 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:10.086720 8835 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:10.094099 9206 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:10.094753 5162 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:10.098878 2147 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:10.101413 2005 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 7 to 8, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:10.112480 4901 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:10.171360 2147 raft_consensus.cc:1203] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:01:10.209025 2405 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00"
dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40858
I20240501 14:01:10.209210 2405 raft_consensus.cc:595] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:10.223132 2147 consensus_queue.cc:237] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:10.225596 5162 raft_consensus.cc:1261] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.226140 5034 raft_consensus.cc:1261] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.226145 9454 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:10.226943 2080 consensus_peers.cc:485] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fac2baeae824f68ae2fe1c62e36d970. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.227123 8870 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.230095 8870 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:10.230803 5032 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:10.234730 5162 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:10.237236 1998 catalog_manager.cc:5484] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:10.259478 2538 consensus_queue.cc:237] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:10.261662 4901 raft_consensus.cc:1261] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.261788 2274 raft_consensus.cc:1261] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:10.261888 2471 consensus_peers.cc:485] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ead586b9a8024a218c55d2bb579a16de. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.262281 8722 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.262589 8633 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.265439 8633 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.266765 2274 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.266770 4901 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.273142 2005 catalog_manager.cc:5484] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.294430 5160 consensus_queue.cc:237] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:10.296980 4901 raft_consensus.cc:1261] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.297077 2274 raft_consensus.cc:1261] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:10.297255 5097 consensus_peers.cc:485] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ed332b3f7304bf69bc348d230ece128. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.297513 9241 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.297822 9174 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.300643 9241 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:10.302742 4901 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:10.304324 2274 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:10.309681 1998 catalog_manager.cc:5484] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:10.329948 2274 consensus_queue.cc:237] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:10.332669 2147 raft_consensus.cc:1261] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.332827 5032 raft_consensus.cc:1261] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:10.333454 2213 consensus_peers.cc:485] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566acdef1d114b8aac9b55ccfa03da35. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.333503 9264 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.333813 9272 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.336405 9264 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:10.336997 5032 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:10.339941 2147 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:10.343075 2005 catalog_manager.cc:5484] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.365650 2537 consensus_queue.cc:237] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
W20240501 14:01:10.369017 2471 consensus_peers.cc:485] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b91e5a14d82419cb295439f577d2786. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.370028 5032 raft_consensus.cc:1261] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.370033 5160 raft_consensus.cc:1261] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.370592 9465 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.370888 8633 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.373698 8722 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:10.379585 5160 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:10.383862 2005 catalog_manager.cc:5484] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.398524 2537 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:10.400061 8829 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:10.401768 5034 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:10.424871 2537 consensus_queue.cc:237] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:10.427855 5032 raft_consensus.cc:1261] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:10.427932 2471 consensus_peers.cc:485] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce213e03db7f4f17913aad989c584607. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.428408 5294 raft_consensus.cc:1261] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.428563 8633 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.428871 9465 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.431725 8722 raft_consensus.cc:2932] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.432386 5032 raft_consensus.cc:2932] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.432883 5294 raft_consensus.cc:2932] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.439601 2000 catalog_manager.cc:5484] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:10.465536 4901 consensus_queue.cc:237] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:10.468304 5160 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.468835 2147 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:10.469988 4834 consensus_peers.cc:485] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 741357494d3a4842935671afd1dbe677. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.470373 9208 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.470685 8871 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.476822 9208 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:10.483609 2147 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:10.487350 5162 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:10.488603 2000 catalog_manager.cc:5484] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.517858 2147 consensus_queue.cc:237] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
W20240501 14:01:10.521279 2082 consensus_peers.cc:485] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e407965bc6047b185913b10dd0681fa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.521396 5294 raft_consensus.cc:1261] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.521440 4901 raft_consensus.cc:1261] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.521983 8701 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.523593 8701 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.525905 8870 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:10.526619 4901 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:10.529356 9429 raft_consensus.cc:979] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: : Instructing follower d1a414d381f44829a2510cb76d481887 to start an election
I20240501 14:01:10.529526 8835 raft_consensus.cc:1067] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Signalling peer d1a414d381f44829a2510cb76d481887 to start an election
I20240501 14:01:10.530192 2146 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00"
dest_uuid: "d1a414d381f44829a2510cb76d481887"
from {username='slave'} at 127.28.17.3:50193
I20240501 14:01:10.530373 2146 raft_consensus.cc:482] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:10.530468 2146 raft_consensus.cc:3037] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:10.532881 2146 raft_consensus.cc:504] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:10.533581 2146 leader_election.cc:288] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:10.534010 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:10.534016 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:01:10.534229 5160 raft_consensus.cc:3037] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:10.534484 4901 raft_consensus.cc:3037] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:10.535260 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd3f8661a13944098a18d3eea7ddeb00" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:10.535456 2405 raft_consensus.cc:3032] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:10.535529 2405 raft_consensus.cc:729] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:01:10.535784 2405 consensus_queue.cc:260] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:10.536193 2405 raft_consensus.cc:3037] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:10.536908 5160 raft_consensus.cc:2454] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:10.536823 2000 catalog_manager.cc:5484] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:10.539563 4901 raft_consensus.cc:2454] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:10.542063 2081 leader_election.cc:302] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:10.542368 9454 raft_consensus.cc:2781] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:10.542520 9454 raft_consensus.cc:686] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:10.542673 2405 raft_consensus.cc:2454] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:10.542788 9454 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:10.547689 2000 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 1 to 2, leader changed from a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.550562 5293 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:10.551276 8829 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:10.555652 9224 raft_consensus.cc:1050] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2: attempting to promote NON_VOTER d1a414d381f44829a2510cb76d481887 to VOTER
I20240501 14:01:10.556473 9224 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:10.573735 2147 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:10.574815 2405 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:10.574944 9434 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:10.576202 9277 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:10.577322 4901 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:10.578039 9277 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:10.589465 5160 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:10.595240 8871 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:10.606734 9527 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.615932 2147 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.623248 4900 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.622888 1998 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.625411 2147 consensus_queue.cc:237] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:10.633006 2408 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.637018 5160 raft_consensus.cc:1261] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.637609 8701 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:10.641111 2080 consensus_peers.cc:485] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a489f3c24c54297a0569dd560cf159e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.645103 5032 raft_consensus.cc:1261] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.645720 8701 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.665848 9532 ts_tablet_manager.cc:902] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:10.665634 5034 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:10.666713 5294 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:10.668061 2405 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:10.669598 9434 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:10.673451 8870 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:10.675005 2147 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:10.675583 9434 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:10.676784 9537 ts_tablet_manager.cc:902] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:10.677665 9434 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.677943 5034 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:10.687577 2145 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.687757 2408 raft_consensus.cc:2932] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.693377 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf051cecef424db5accfce66623346b5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:10.693723 2000 catalog_manager.cc:5484] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.696735 2006 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:10.701087 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet bf051cecef424db5accfce66623346b5 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:10.721554 9538 tablet_replica.cc:330] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:10.723006 9538 raft_consensus.cc:2229] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:10.723376 5162 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:10.724339 9538 raft_consensus.cc:2258] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:10.726688 9538 ts_tablet_manager.cc:1875] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:10.735234 9532 tablet_copy_client.cc:308] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:01:10.735926 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet 0fac2baeae824f68ae2fe1c62e36d970 from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:38549)
I20240501 14:01:10.736094 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet 0fac2baeae824f68ae2fe1c62e36d970 from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:38549: session id = 745ac12076e84effa5bde1ecbd6a32ec-0fac2baeae824f68ae2fe1c62e36d970
I20240501 14:01:10.739058 2167 tablet_copy_source_session.cc:205] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:10.742197 9532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fac2baeae824f68ae2fe1c62e36d970. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.745687 9537 tablet_copy_client.cc:308] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:10.745684 9538 ts_tablet_manager.cc:1888] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:01:10.745833 9538 log.cc:1196] T bf051cecef424db5accfce66623346b5 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/bf051cecef424db5accfce66623346b5
I20240501 14:01:10.746436 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet ead586b9a8024a218c55d2bb579a16de from peer d1a414d381f44829a2510cb76d481887 ({username='slave'} at 127.28.17.1:53971)
I20240501 14:01:10.746596 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet ead586b9a8024a218c55d2bb579a16de from peer d1a414d381f44829a2510cb76d481887 at {username='slave'} at 127.28.17.1:53971: session id = d1a414d381f44829a2510cb76d481887-ead586b9a8024a218c55d2bb579a16de
I20240501 14:01:10.747632 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet bf051cecef424db5accfce66623346b5 (table rebalancer_extra_table_6 [id=19dc643384eb44d39450af8eff851146]) successfully deleted
I20240501 14:01:10.749725 2558 tablet_copy_source_session.cc:205] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:10.752740 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead586b9a8024a218c55d2bb579a16de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.759658 9537 tablet_copy_client.cc:783] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:10.759898 9537 tablet_copy_client.cc:647] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:10.762178 9532 tablet_copy_client.cc:783] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:10.762421 9532 tablet_copy_client.cc:647] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:10.764936 9537 tablet_copy_client.cc:523] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:10.767760 5034 consensus_queue.cc:237] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:10.769380 9537 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:10.769496 9532 tablet_copy_client.cc:523] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:10.776798 9532 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
W20240501 14:01:10.780511 4968 consensus_peers.cc:485] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f647f3f27faf435f81e076bfc2ef0795. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.780620 2538 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.781754 9434 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.782145 9537 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:10.782558 9537 tablet_bootstrap.cc:492] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: Bootstrap complete.
I20240501 14:01:10.782810 9537 ts_tablet_manager.cc:1367] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20240501 14:01:10.783964 9537 raft_consensus.cc:348] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:10.784190 9537 raft_consensus.cc:729] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: LEARNER
I20240501 14:01:10.784384 9537 consensus_queue.cc:260] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:10.787484 9537 ts_tablet_manager.cc:1398] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:01:10.789641 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session d1a414d381f44829a2510cb76d481887-ead586b9a8024a218c55d2bb579a16de received from {username='slave'} at 127.28.17.1:53971
I20240501 14:01:10.789754 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session d1a414d381f44829a2510cb76d481887-ead586b9a8024a218c55d2bb579a16de on tablet ead586b9a8024a218c55d2bb579a16de with peer d1a414d381f44829a2510cb76d481887
I20240501 14:01:10.791157 9532 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:10.791520 9532 tablet_bootstrap.cc:492] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:10.791769 9532 ts_tablet_manager.cc:1367] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240501 14:01:10.797374 9532 raft_consensus.cc:348] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:10.797636 9532 raft_consensus.cc:729] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:10.797817 9532 consensus_queue.cc:260] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:10.799716 9556 ts_tablet_manager.cc:902] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:10.800626 9532 ts_tablet_manager.cc:1398] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 14:01:10.801599 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-0fac2baeae824f68ae2fe1c62e36d970 received from {username='slave'} at 127.28.17.4:38549
I20240501 14:01:10.801705 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-0fac2baeae824f68ae2fe1c62e36d970 on tablet 0fac2baeae824f68ae2fe1c62e36d970 with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:10.805380 5162 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:10.806083 9527 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:10.805740 9434 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:10.811913 9552 ts_tablet_manager.cc:902] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:10.827885 2538 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:10.830703 5162 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:10.833165 9552 tablet_copy_client.cc:308] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:01:10.833523 1998 catalog_manager.cc:5484] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.833949 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 566acdef1d114b8aac9b55ccfa03da35 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:39671)
I20240501 14:01:10.834127 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 566acdef1d114b8aac9b55ccfa03da35 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:39671: session id = 5a4b9a96e9734baa8a03d6056bfa934c-566acdef1d114b8aac9b55ccfa03da35
I20240501 14:01:10.837025 2298 tablet_copy_source_session.cc:205] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:10.841462 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566acdef1d114b8aac9b55ccfa03da35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.848506 9556 tablet_copy_client.cc:308] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:10.848963 9552 tablet_copy_client.cc:783] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:10.849251 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 4ed332b3f7304bf69bc348d230ece128 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:52851)
I20240501 14:01:10.849303 9552 tablet_copy_client.cc:647] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:10.849442 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 4ed332b3f7304bf69bc348d230ece128 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:52851: session id = a64f34785a7b44fa8070b0452a4e0ddd-4ed332b3f7304bf69bc348d230ece128
I20240501 14:01:10.852193 5184 tablet_copy_source_session.cc:205] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:10.852530 9552 tablet_copy_client.cc:523] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:10.853688 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed332b3f7304bf69bc348d230ece128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.855544 2538 consensus_queue.cc:237] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:10.858448 9552 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:10.862363 9556 tablet_copy_client.cc:783] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:10.862599 9556 tablet_copy_client.cc:647] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:10.867362 9556 tablet_copy_client.cc:523] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:10.871354 9556 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:10.903584 9575 ts_tablet_manager.cc:902] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:10.908581 9556 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:10.909077 9556 tablet_bootstrap.cc:492] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:01:10.909389 9556 ts_tablet_manager.cc:1367] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.004s
I20240501 14:01:10.910089 9556 raft_consensus.cc:348] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:10.910482 9556 raft_consensus.cc:729] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:01:10.910743 9556 consensus_queue.cc:260] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:10.911720 9556 ts_tablet_manager.cc:1398] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:10.912860 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-4ed332b3f7304bf69bc348d230ece128 received from {username='slave'} at 127.28.17.3:52851
I20240501 14:01:10.913009 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-4ed332b3f7304bf69bc348d230ece128 on tablet 4ed332b3f7304bf69bc348d230ece128 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:10.915093 9537 ts_tablet_manager.cc:902] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:10.921931 4901 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:10.922686 8722 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:01:10.925554 2473 consensus_peers.cc:485] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cb2dab6f4684bb18bddc9edba1d6517. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:10.927264 2145 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:10.927862 8722 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:10.931201 9465 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:10.935361 4900 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:10.949096 1998 catalog_manager.cc:5484] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.955554 9552 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:10.959056 9552 tablet_bootstrap.cc:492] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:10.960510 9552 ts_tablet_manager.cc:1367] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.102s user 0.024s sys 0.027s
I20240501 14:01:10.962172 9552 raft_consensus.cc:348] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:10.962455 9552 raft_consensus.cc:729] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:01:10.962693 9552 consensus_queue.cc:260] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:10.964766 9556 ts_tablet_manager.cc:902] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:10.965044 9552 ts_tablet_manager.cc:1398] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:10.966257 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-566acdef1d114b8aac9b55ccfa03da35 received from {username='slave'} at 127.28.17.8:39671
I20240501 14:01:10.966408 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-566acdef1d114b8aac9b55ccfa03da35 on tablet 566acdef1d114b8aac9b55ccfa03da35 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:01:10.974795 2146 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:10.985739 9556 tablet_copy_client.cc:308] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:01:10.986526 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet 741357494d3a4842935671afd1dbe677 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:36677)
I20240501 14:01:10.986711 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet 741357494d3a4842935671afd1dbe677 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:36677: session id = a64f34785a7b44fa8070b0452a4e0ddd-741357494d3a4842935671afd1dbe677
I20240501 14:01:10.986829 9575 tablet_copy_client.cc:308] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:10.987571 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 9b91e5a14d82419cb295439f577d2786 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35609)
I20240501 14:01:10.987713 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 9b91e5a14d82419cb295439f577d2786 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35609: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-9b91e5a14d82419cb295439f577d2786
I20240501 14:01:10.989648 4921 tablet_copy_source_session.cc:205] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:10.990403 2558 tablet_copy_source_session.cc:205] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:10.991168 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 741357494d3a4842935671afd1dbe677. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.994624 9575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b91e5a14d82419cb295439f577d2786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:10.998872 9556 tablet_copy_client.cc:783] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:10.999089 9556 tablet_copy_client.cc:647] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.001387 9556 tablet_copy_client.cc:523] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.004257 9556 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:11.009151 9556 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.009529 9556 tablet_bootstrap.cc:492] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:01:11.009786 9556 ts_tablet_manager.cc:1367] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:01:11.010605 9556 raft_consensus.cc:348] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:11.010823 9556 raft_consensus.cc:729] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:01:11.010995 9556 consensus_queue.cc:260] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:11.011725 9556 ts_tablet_manager.cc:1398] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:11.016028 9575 tablet_copy_client.cc:783] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:11.016569 9575 tablet_copy_client.cc:647] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.017530 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-741357494d3a4842935671afd1dbe677 received from {username='slave'} at 127.28.17.3:36677
I20240501 14:01:11.017689 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-741357494d3a4842935671afd1dbe677 on tablet 741357494d3a4842935671afd1dbe677 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:11.022293 9596 ts_tablet_manager.cc:902] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:11.018435 2408 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:11.023926 9515 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:11.026322 9575 tablet_copy_client.cc:523] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.030225 9537 tablet_copy_client.cc:308] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:11.030908 9575 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:11.030966 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet ce213e03db7f4f17913aad989c584607 from peer d1a414d381f44829a2510cb76d481887 ({username='slave'} at 127.28.17.1:53971)
I20240501 14:01:11.031137 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet ce213e03db7f4f17913aad989c584607 from peer d1a414d381f44829a2510cb76d481887 at {username='slave'} at 127.28.17.1:53971: session id = d1a414d381f44829a2510cb76d481887-ce213e03db7f4f17913aad989c584607
I20240501 14:01:11.033282 2558 tablet_copy_source_session.cc:205] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.034566 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce213e03db7f4f17913aad989c584607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.040910 9537 tablet_copy_client.cc:783] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:11.041178 9537 tablet_copy_client.cc:647] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.042810 9537 tablet_copy_client.cc:523] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.044163 9596 tablet_copy_client.cc:308] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:01:11.044932 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet 0a489f3c24c54297a0569dd560cf159e from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:41619)
I20240501 14:01:11.045099 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet 0a489f3c24c54297a0569dd560cf159e from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:41619: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-0a489f3c24c54297a0569dd560cf159e
I20240501 14:01:11.047297 9537 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:11.047780 2167 tablet_copy_source_session.cc:205] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.049955 9596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a489f3c24c54297a0569dd560cf159e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.052858 2538 raft_consensus.cc:1203] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.053120 9537 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.053439 9537 tablet_bootstrap.cc:492] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: Bootstrap complete.
I20240501 14:01:11.053655 9537 ts_tablet_manager.cc:1367] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240501 14:01:11.055596 9537 raft_consensus.cc:348] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:11.055833 9537 raft_consensus.cc:729] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: LEARNER
I20240501 14:01:11.055994 9537 consensus_queue.cc:260] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:11.056627 9537 ts_tablet_manager.cc:1398] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:01:11.057705 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session d1a414d381f44829a2510cb76d481887-ce213e03db7f4f17913aad989c584607 received from {username='slave'} at 127.28.17.1:53971
I20240501 14:01:11.057813 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session d1a414d381f44829a2510cb76d481887-ce213e03db7f4f17913aad989c584607 on tablet ce213e03db7f4f17913aad989c584607 with peer d1a414d381f44829a2510cb76d481887
I20240501 14:01:11.059929 5160 consensus_queue.cc:237] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:11.062752 9596 tablet_copy_client.cc:783] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:11.063118 9596 tablet_copy_client.cc:647] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.065542 9575 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.065848 9596 tablet_copy_client.cc:523] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.065970 9575 tablet_bootstrap.cc:492] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:11.066251 9575 ts_tablet_manager.cc:1367] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.004s
W20240501 14:01:11.067800 5099 consensus_peers.cc:485] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdb2924e91fb4a98aaa6bf977f682fdb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:11.068156 2408 raft_consensus.cc:1261] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.068336 2277 raft_consensus.cc:1261] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.068295 9575 raft_consensus.cc:348] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:11.068552 9575 raft_consensus.cc:729] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:11.068775 9575 consensus_queue.cc:260] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:11.068981 9241 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.069552 9575 ts_tablet_manager.cc:1398] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:11.070775 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-9b91e5a14d82419cb295439f577d2786 received from {username='slave'} at 127.28.17.2:35609
I20240501 14:01:11.070825 4900 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:11.070895 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-9b91e5a14d82419cb295439f577d2786 on tablet 9b91e5a14d82419cb295439f577d2786 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:11.071195 9596 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:11.071240 9390 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.072750 8701 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:11.079699 5160 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:01:11.080350 9515 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:11.091071 4901 consensus_queue.cc:237] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:11.093657 2538 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:11.094393 9208 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:11.097533 9208 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:11.099141 2538 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:11.108083 5160 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:11.109014 8829 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:11.111481 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9eec5cc39eda4f6d85f83959952a322c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:11.113596 5160 raft_consensus.cc:2932] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:11.113338 1998 catalog_manager.cc:5484] T 9eec5cc39eda4f6d85f83959952a322c P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 5, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:11.119648 9390 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.123343 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eec5cc39eda4f6d85f83959952a322c with delete_type TABLET_DATA_TOMBSTONED (TS 5a4b9a96e9734baa8a03d6056bfa934c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36154
I20240501 14:01:11.126849 2274 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.128769 9613 tablet_replica.cc:330] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:01:11.128989 9613 raft_consensus.cc:2229] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:11.129220 9613 raft_consensus.cc:2258] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:11.133438 9613 ts_tablet_manager.cc:1875] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:11.133800 2006 catalog_manager.cc:5484] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.146737 9611 ts_tablet_manager.cc:902] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:11.154382 2407 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.169358 9596 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.169903 9596 tablet_bootstrap.cc:492] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:11.170270 9596 ts_tablet_manager.cc:1367] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.099s user 0.021s sys 0.022s
I20240501 14:01:11.171182 9596 raft_consensus.cc:348] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:11.173130 9596 raft_consensus.cc:729] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:11.174461 9596 consensus_queue.cc:260] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:11.177043 9596 ts_tablet_manager.cc:1398] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:11.177510 9613 ts_tablet_manager.cc:1888] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 14:01:11.178252 9613 log.cc:1196] T 9eec5cc39eda4f6d85f83959952a322c P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/9eec5cc39eda4f6d85f83959952a322c
I20240501 14:01:11.178246 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-0a489f3c24c54297a0569dd560cf159e received from {username='slave'} at 127.28.17.2:41619
I20240501 14:01:11.178376 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-0a489f3c24c54297a0569dd560cf159e on tablet 0a489f3c24c54297a0569dd560cf159e with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:11.190869 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 9eec5cc39eda4f6d85f83959952a322c (table rebalancer_extra_table_34 [id=771bbd383da94f96882a19a150c9e550]) successfully deleted
I20240501 14:01:11.192729 2146 consensus_queue.cc:237] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:11.195143 5160 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:11.195233 4900 raft_consensus.cc:1261] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:01:11.195797 9515 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:11.196291 9515 consensus_queue.cc:1035] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:01:11.198750 8701 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:11.199378 4900 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:11.205145 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd3f8661a13944098a18d3eea7ddeb00 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:11.205122 5162 raft_consensus.cc:2932] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:11.210453 2000 catalog_manager.cc:5484] T fd3f8661a13944098a18d3eea7ddeb00 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 8 to 10, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:11.216097 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet fd3f8661a13944098a18d3eea7ddeb00 with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51622
I20240501 14:01:11.237838 2146 raft_consensus.cc:1203] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.248474 9623 tablet_replica.cc:330] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:01:11.250165 9623 raft_consensus.cc:2229] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:11.250707 9623 raft_consensus.cc:2258] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:11.252074 9623 ts_tablet_manager.cc:1875] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:11.257777 5294 consensus_queue.cc:237] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:11.271338 9611 tablet_copy_client.cc:308] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:01:11.272032 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet 0e407965bc6047b185913b10dd0681fa from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:50703)
I20240501 14:01:11.272186 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet 0e407965bc6047b185913b10dd0681fa from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:50703: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-0e407965bc6047b185913b10dd0681fa
I20240501 14:01:11.272943 5162 raft_consensus.cc:1261] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.273347 9623 ts_tablet_manager.cc:1888] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:01:11.273487 9623 log.cc:1196] T fd3f8661a13944098a18d3eea7ddeb00 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd3f8661a13944098a18d3eea7ddeb00
I20240501 14:01:11.274246 5031 raft_consensus.cc:1261] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.274405 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet fd3f8661a13944098a18d3eea7ddeb00 (table rebalance_test_table_3 [id=76b7d7614f024679b51f64d8859e02ae]) successfully deleted
I20240501 14:01:11.274849 2167 tablet_copy_source_session.cc:205] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.274940 9400 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.275480 9400 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.276764 9611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e407965bc6047b185913b10dd0681fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.279183 2277 raft_consensus.cc:1203] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.285434 9611 tablet_copy_client.cc:783] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:11.285799 9611 tablet_copy_client.cc:647] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.287464 9611 tablet_copy_client.cc:523] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.289400 5294 raft_consensus.cc:1203] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.292551 9611 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
W20240501 14:01:11.299376 5228 consensus_peers.cc:485] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b911b0ff6ef446d8a6637edca57e570d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:11.304198 9400 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:11.307543 5162 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:11.310555 9611 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.310163 5032 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:11.311159 9611 tablet_bootstrap.cc:492] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:01:11.311543 9611 ts_tablet_manager.cc:1367] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20240501 14:01:11.312633 9611 raft_consensus.cc:348] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:11.312891 9611 raft_consensus.cc:729] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:01:11.313127 9611 consensus_queue.cc:260] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:11.313850 9611 ts_tablet_manager.cc:1398] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:11.314816 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-0e407965bc6047b185913b10dd0681fa received from {username='slave'} at 127.28.17.6:50703
I20240501 14:01:11.314924 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-0e407965bc6047b185913b10dd0681fa on tablet 0e407965bc6047b185913b10dd0681fa with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:11.316305 1998 catalog_manager.cc:5484] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:11.337092 9637 ts_tablet_manager.cc:902] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:11.344318 5293 consensus_queue.cc:237] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:11.345561 9637 tablet_copy_client.cc:308] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.6:46037
I20240501 14:01:11.346382 5054 tablet_copy_service.cc:140] P 6f81f803ec7746ccb502c59b7cbf8eb2: Received BeginTabletCopySession request for tablet f647f3f27faf435f81e076bfc2ef0795 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:54863)
I20240501 14:01:11.346570 5054 tablet_copy_service.cc:161] P 6f81f803ec7746ccb502c59b7cbf8eb2: Beginning new tablet copy session on tablet f647f3f27faf435f81e076bfc2ef0795 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:54863: session id = f8c60f2bbac541dcba4424ab726f9ad9-f647f3f27faf435f81e076bfc2ef0795
I20240501 14:01:11.346964 5162 raft_consensus.cc:1261] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:11.347196 5228 consensus_peers.cc:485] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: beb5db06e4b74430b2bcbbecc82796e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:11.348251 4900 raft_consensus.cc:1261] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.348528 9400 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.349332 9400 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.349576 5054 tablet_copy_source_session.cc:205] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.356976 9637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647f3f27faf435f81e076bfc2ef0795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.368497 9631 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:11.370589 5162 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:11.373309 2406 raft_consensus.cc:1203] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.374792 9637 tablet_copy_client.cc:783] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:11.375278 9637 tablet_copy_client.cc:647] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.377487 9637 tablet_copy_client.cc:523] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.383634 9637 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:11.386973 1998 catalog_manager.cc:5484] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:11.388984 4900 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:11.397454 9552 ts_tablet_manager.cc:902] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:11.421908 9637 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.422536 9637 tablet_bootstrap.cc:492] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:01:11.422818 9637 ts_tablet_manager.cc:1367] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.017s
I20240501 14:01:11.423679 9637 raft_consensus.cc:348] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:11.423941 9637 raft_consensus.cc:729] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:01:11.424192 9637 consensus_queue.cc:260] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:11.424794 9637 ts_tablet_manager.cc:1398] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:11.427146 5054 tablet_copy_service.cc:342] P 6f81f803ec7746ccb502c59b7cbf8eb2: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-f647f3f27faf435f81e076bfc2ef0795 received from {username='slave'} at 127.28.17.5:54863
I20240501 14:01:11.427366 5054 tablet_copy_service.cc:433] P 6f81f803ec7746ccb502c59b7cbf8eb2: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-f647f3f27faf435f81e076bfc2ef0795 on tablet f647f3f27faf435f81e076bfc2ef0795 with peer f8c60f2bbac541dcba4424ab726f9ad9
W20240501 14:01:11.428862 2343 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.8:34221, user_credentials={real_user=slave}} blocked reactor thread for 47760.3us
I20240501 14:01:11.431735 9552 tablet_copy_client.cc:308] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:11.433197 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 1cb2dab6f4684bb18bddc9edba1d6517 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:41611)
I20240501 14:01:11.433363 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 1cb2dab6f4684bb18bddc9edba1d6517 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:41611: session id = 5a4b9a96e9734baa8a03d6056bfa934c-1cb2dab6f4684bb18bddc9edba1d6517
I20240501 14:01:11.436122 2558 tablet_copy_source_session.cc:205] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.437539 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb2dab6f4684bb18bddc9edba1d6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.449129 9552 tablet_copy_client.cc:783] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:11.449436 9552 tablet_copy_client.cc:647] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.453842 9552 tablet_copy_client.cc:523] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.454262 2145 consensus_queue.cc:237] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:11.457569 9552 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:11.464893 2406 raft_consensus.cc:1203] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:01:11.471056 2082 consensus_peers.cc:485] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3fe9fe1c13743478e09f5eeeb528b57. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:11.471735 2537 raft_consensus.cc:1261] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.472463 8870 consensus_queue.cc:1035] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.475749 2406 raft_consensus.cc:1261] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:11.476320 8870 consensus_queue.cc:1035] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:11.492399 9515 raft_consensus.cc:2932] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.493119 2406 raft_consensus.cc:2932] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.498405 2146 raft_consensus.cc:1203] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.504142 1998 catalog_manager.cc:5484] T e3fe9fe1c13743478e09f5eeeb528b57 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.507419 2538 raft_consensus.cc:2932] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:11.509919 2277 raft_consensus.cc:1203] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.555562 9552 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.556188 9552 tablet_bootstrap.cc:492] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:11.556489 9552 ts_tablet_manager.cc:1367] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.099s user 0.046s sys 0.030s
I20240501 14:01:11.559048 9552 raft_consensus.cc:348] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:11.559365 9552 raft_consensus.cc:729] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:01:11.559588 9552 consensus_queue.cc:260] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:11.560832 9552 ts_tablet_manager.cc:1398] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:11.561925 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-1cb2dab6f4684bb18bddc9edba1d6517 received from {username='slave'} at 127.28.17.8:41611
I20240501 14:01:11.562034 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-1cb2dab6f4684bb18bddc9edba1d6517 on tablet 1cb2dab6f4684bb18bddc9edba1d6517 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:01:11.626156 9515 raft_consensus.cc:1050] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:11.627072 9515 consensus_queue.cc:237] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:11.631278 5031 raft_consensus.cc:1261] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.631336 2537 raft_consensus.cc:1261] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.631772 8870 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.631826 5162 raft_consensus.cc:1261] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.635429 8701 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.635702 8870 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.646716 9611 ts_tablet_manager.cc:902] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:11.667299 9465 raft_consensus.cc:1050] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER d1a414d381f44829a2510cb76d481887 to VOTER
I20240501 14:01:11.668267 9465 consensus_queue.cc:237] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:11.674247 5031 raft_consensus.cc:1203] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.694348 2277 raft_consensus.cc:1261] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.696097 8722 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.698906 4900 raft_consensus.cc:1261] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.700199 8722 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.700381 2146 raft_consensus.cc:1261] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.702243 8633 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.705070 8722 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.708091 4901 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.723181 2277 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.727928 8870 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.728240 2005 catalog_manager.cc:5484] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:11.735515 2537 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.741434 9264 raft_consensus.cc:1050] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:11.741722 5031 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.742303 9264 consensus_queue.cc:237] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:11.748800 2537 consensus_queue.cc:237] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:11.751256 1998 catalog_manager.cc:5484] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.754284 9465 raft_consensus.cc:1050] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:11.755193 9465 consensus_queue.cc:237] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:11.760206 5162 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.761415 9611 tablet_copy_client.cc:308] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:11.762665 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet fdb2924e91fb4a98aaa6bf977f682fdb from peer 6f81f803ec7746ccb502c59b7cbf8eb2 ({username='slave'} at 127.28.17.6:54429)
I20240501 14:01:11.762826 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet fdb2924e91fb4a98aaa6bf977f682fdb from peer 6f81f803ec7746ccb502c59b7cbf8eb2 at {username='slave'} at 127.28.17.6:54429: session id = 6f81f803ec7746ccb502c59b7cbf8eb2-fdb2924e91fb4a98aaa6bf977f682fdb
I20240501 14:01:11.765898 5184 tablet_copy_source_session.cc:205] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.737259 2146 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.771121 9611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb2924e91fb4a98aaa6bf977f682fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.779986 5294 raft_consensus.cc:1261] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.783051 5031 raft_consensus.cc:1261] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.784155 9465 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.785486 9716 ts_tablet_manager.cc:902] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:11.785392 9271 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.786078 2274 raft_consensus.cc:1261] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.787130 5161 raft_consensus.cc:1261] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.788164 8633 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.788710 8722 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.795959 9611 tablet_copy_client.cc:783] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:11.796320 9611 tablet_copy_client.cc:647] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:11.798043 9611 tablet_copy_client.cc:523] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:11.801738 4901 raft_consensus.cc:1261] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:11.802695 8722 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:11.805264 9611 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap starting.
I20240501 14:01:11.805553 2146 raft_consensus.cc:1261] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:11.806025 4901 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.805025 9465 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.808156 8633 consensus_queue.cc:1035] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:11.813508 2146 raft_consensus.cc:2932] T ead586b9a8024a218c55d2bb579a16de P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.816921 4900 raft_consensus.cc:1203] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:11.818275 2145 raft_consensus.cc:1261] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.818799 5032 raft_consensus.cc:1261] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.819424 9264 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.820376 9264 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.824227 9727 ts_tablet_manager.cc:902] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:11.841840 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ead586b9a8024a218c55d2bb579a16de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:11.841382 2006 catalog_manager.cc:5484] T ead586b9a8024a218c55d2bb579a16de P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:11.850766 9271 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:11.857983 5031 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:11.859274 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet ead586b9a8024a218c55d2bb579a16de with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35676
I20240501 14:01:11.860307 9390 raft_consensus.cc:1050] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:01:11.861303 9390 consensus_queue.cc:237] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:11.863971 2146 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:11.873912 2005 catalog_manager.cc:5484] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:11.915426 2145 consensus_queue.cc:237] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:11.921844 2537 raft_consensus.cc:1261] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:11.922423 9696 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:11.925287 9696 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.929483 2537 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.933807 9735 tablet_replica.cc:330] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:01:11.934115 9735 raft_consensus.cc:2229] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:11.934547 9735 raft_consensus.cc:2258] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:11.935577 9465 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:11.936229 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fac2baeae824f68ae2fe1c62e36d970 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:11.936713 5162 raft_consensus.cc:1261] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:11.937219 9696 consensus_queue.cc:1035] T 0fac2baeae824f68ae2fe1c62e36d970 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:11.938655 5160 raft_consensus.cc:2932] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.937940 5161 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:11.943255 5032 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:11.944330 2005 catalog_manager.cc:5484] T 0fac2baeae824f68ae2fe1c62e36d970 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:11.949012 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fac2baeae824f68ae2fe1c62e36d970 with delete_type TABLET_DATA_TOMBSTONED (TS 6f81f803ec7746ccb502c59b7cbf8eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58280
I20240501 14:01:11.951872 9208 raft_consensus.cc:1050] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:01:11.952759 9208 consensus_queue.cc:237] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:11.955065 5296 raft_consensus.cc:1203] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:11.959733 9735 ts_tablet_manager.cc:1875] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:11.963421 9716 tablet_copy_client.cc:308] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:01:11.963277 5294 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:11.964051 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet b911b0ff6ef446d8a6637edca57e570d from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35917)
I20240501 14:01:11.964813 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet b911b0ff6ef446d8a6637edca57e570d from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35917: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-b911b0ff6ef446d8a6637edca57e570d
I20240501 14:01:11.967730 5316 tablet_copy_source_session.cc:205] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.968798 2146 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.968945 2406 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.969528 8871 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.970474 9716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b911b0ff6ef446d8a6637edca57e570d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.971544 8871 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.970014 5162 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.973234 8871 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.973407 9727 tablet_copy_client.cc:308] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:01:11.970185 2274 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:11.975049 4901 raft_consensus.cc:1261] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.975685 9609 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.975478 8633 raft_consensus.cc:1050] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER d1a414d381f44829a2510cb76d481887 to VOTER
I20240501 14:01:11.977595 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet beb5db06e4b74430b2bcbbecc82796e2 from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:35953)
I20240501 14:01:11.977808 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet beb5db06e4b74430b2bcbbecc82796e2 from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:35953: session id = 745ac12076e84effa5bde1ecbd6a32ec-beb5db06e4b74430b2bcbbecc82796e2
I20240501 14:01:11.978032 8633 consensus_queue.cc:237] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:11.979488 2406 raft_consensus.cc:1261] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.985195 9611 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:11.985689 9611 tablet_bootstrap.cc:492] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: Bootstrap complete.
I20240501 14:01:11.985989 9611 ts_tablet_manager.cc:1367] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent bootstrapping tablet: real 0.181s user 0.008s sys 0.016s
I20240501 14:01:11.986872 5316 tablet_copy_source_session.cc:205] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:11.986950 9611 raft_consensus.cc:348] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:11.987254 9611 raft_consensus.cc:729] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f81f803ec7746ccb502c59b7cbf8eb2, State: Initialized, Role: LEARNER
I20240501 14:01:11.987483 9611 consensus_queue.cc:260] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:11.989187 9727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5db06e4b74430b2bcbbecc82796e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:11.991333 2277 raft_consensus.cc:1261] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.994398 9609 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:11.998059 5296 raft_consensus.cc:1261] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:11.999402 9749 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.000368 5031 raft_consensus.cc:1261] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.000768 9740 tablet_replica.cc:330] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:01:12.000676 8871 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.001443 8633 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.005071 9716 tablet_copy_client.cc:783] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:12.005336 9716 tablet_copy_client.cc:647] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:12.008391 5162 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.007696 9716 tablet_copy_client.cc:523] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:12.016752 9716 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:12.018719 2145 raft_consensus.cc:1261] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.020829 2146 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.021448 9727 tablet_copy_client.cc:783] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:12.021704 9727 tablet_copy_client.cc:647] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:12.024497 2000 catalog_manager.cc:5484] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.025957 9609 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.026928 8633 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.034168 9740 raft_consensus.cc:2229] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.034474 9740 raft_consensus.cc:2258] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.039821 9753 ts_tablet_manager.cc:902] T e3fe9fe1c13743478e09f5eeeb528b57 P 6f81f803ec7746ccb502c59b7cbf8eb2: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:12.040694 9727 tablet_copy_client.cc:523] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
F20240501 14:01:12.040848 9753 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-01096bd6f2bf4663b70043f61ecdea5b.queue_time_us, tablet-download-pool-01096bd6f2bf4663b70043f61ecdea5b.queue_time_us) (tablet-download-pool-01096bd6f2bf4663b70043f61ecdea5b.run_cpu_time_us, tablet-download-pool-01096bd6f2bf4663b70043f61ecdea5b.run_cpu_time_us) (tablet-download-pool-01096bd6f2bf4663b70043f61ecdea5b.run_wall_time_us, tablet-download-pool-01096bd6f2bf4663b70043f61ecdea5b.run_wall_time_us) (tablet-download-pool-0e407965bc6047b185913b10dd0681fa.queue_time_us, tablet-download-pool-0e407965bc6047b185913b10dd0681fa.queue_time_us) (tablet-download-pool-0e407965bc6047b185913b10dd0681fa.run_cpu_time_us, tablet-download-pool-0e407965bc6047b185913b10dd0681fa.run_cpu_time_us) (tablet-download-pool-0e407965bc6047b185913b10dd0681fa.run_wall_time_us, tablet-download-pool-0e407965bc6047b185913b10dd0681fa.run_wall_time_us) (tablet-download-pool-28f67dedd5f44301b00d3e9c08330198.queue_time_us, tablet-download-pool-28f67dedd5f44301b00d3e9c08330198.queue_time_us) (tablet-download-pool-28f67dedd5f44301b00d3e9c08330198.run_cpu_time_us, tablet-download-pool-28f67dedd5f44301b00d3e9c08330198.run_cpu_time_us) (tablet-download-pool-28f67dedd5f44301b00d3e9c08330198.run_wall_time_us, tablet-download-pool-28f67dedd5f44301b00d3e9c08330198.run_wall_time_us) (tablet-download-pool-2ee13db8a347476fbdabdb402c93b584.queue_time_us, tablet-download-pool-2ee13db8a347476fbdabdb402c93b584.queue_time_us) (tablet-download-pool-2ee13db8a347476fbdabdb402c93b584.run_cpu_time_us, tablet-download-pool-2ee13db8a347476fbdabdb402c93b584.run_cpu_time_us) (tablet-download-pool-2ee13db8a347476fbdabdb402c93b584.run_wall_time_us, tablet-download-pool-2ee13db8a347476fbdabdb402c93b584.run_wall_time_us) (tablet-download-pool-387698a371cc42eab970f2f4471bbf5b.queue_time_us, tablet-download-pool-387698a371cc42eab970f2f4471bbf5b.queue_time_us) (tablet-download-pool-387698a371cc42eab970f2f4471bbf5b.run_cpu_time_us, tablet-download-pool-387698a371cc42eab970f2f4471bbf5b.run_cpu_time_us) (tablet-download-pool-387698a371cc42eab970f2f4471bbf5b.run_wall_time_us, tablet-download-pool-387698a371cc42eab970f2f4471bbf5b.run_wall_time_us) (tablet-download-pool-44a4e92b144f47b5800506f4bc3eadb6.queue_time_us, tablet-download-pool-44a4e92b144f47b5800506f4bc3eadb6.queue_time_us) (tablet-download-pool-44a4e92b144f47b5800506f4bc3eadb6.run_cpu_time_us, tablet-download-pool-44a4e92b144f47b5800506f4bc3eadb6.run_cpu_time_us) (tablet-download-pool-44a4e92b144f47b5800506f4bc3eadb6.run_wall_time_us, tablet-download-pool-44a4e92b144f47b5800506f4bc3eadb6.run_wall_time_us) (tablet-download-pool-457c62e1ddf547ad900ed529b3244474.queue_time_us, tablet-download-pool-457c62e1ddf547ad900ed529b3244474.queue_time_us) (tablet-download-pool-457c62e1ddf547ad900ed529b3244474.run_cpu_time_us, tablet-download-pool-457c62e1ddf547ad900ed529b3244474.run_cpu_time_us) (tablet-download-pool-457c62e1ddf547ad900ed529b3244474.run_wall_time_us, tablet-download-pool-457c62e1ddf547ad900ed529b3244474.run_wall_time_us) (tablet-download-pool-5b1f151470f24501857f38461d5b70c8.queue_time_us, tablet-download-pool-5b1f151470f24501857f38461d5b70c8.queue_time_us) (tablet-download-pool-5b1f151470f24501857f38461d5b70c8.run_cpu_time_us, tablet-download-pool-5b1f151470f24501857f38461d5b70c8.run_cpu_time_us) (tablet-download-pool-5b1f151470f24501857f38461d5b70c8.run_wall_time_us, tablet-download-pool-5b1f151470f24501857f38461d5b70c8.run_wall_time_us) (tablet-download-pool-6c74bacbf4464bab9c403e2153953474.queue_time_us, tablet-download-pool-6c74bacbf4464bab9c403e2153953474.queue_time_us) (tablet-download-pool-6c74bacbf4464bab9c403e2153953474.run_cpu_time_us, tablet-download-pool-6c74bacbf4464bab9c403e2153953474.run_cpu_time_us) (tablet-download-pool-6c74bacbf4464bab9c403e2153953474.run_wall_time_us, tablet-download-pool-6c74bacbf4464bab9c403e2153953474.run_wall_time_us) (tablet-download-pool-83392c08bf314fcd98833335c247ccb1.queue_time_us, tablet-download-pool-83392c08bf314fcd98833335c247ccb1.queue_time_us) (tablet-download-pool-83392c08bf314fcd98833335c247ccb1.run_cpu_time_us, tablet-download-pool-83392c08bf314fcd98833335c247ccb1.run_cpu_time_us) (tablet-download-pool-83392c08bf314fcd98833335c247ccb1.run_wall_time_us, tablet-download-pool-83392c08bf314fcd98833335c247ccb1.run_wall_time_us) (tablet-download-pool-d42a56dd51fe410cb42dd284ecbf2748.queue_time_us, tablet-download-pool-d42a56dd51fe410cb42dd284ecbf2748.queue_time_us) (tablet-download-pool-d42a56dd51fe410cb42dd284ecbf2748.run_cpu_time_us, tablet-download-pool-d42a56dd51fe410cb42dd284ecbf2748.run_cpu_time_us) (tablet-download-pool-d42a56dd51fe410cb42dd284ecbf2748.run_wall_time_us, tablet-download-pool-d42a56dd51fe410cb42dd284ecbf2748.run_wall_time_us) (tablet-download-pool-e3fe9fe1c13743478e09f5eeeb528b57.queue_time_us, tablet-download-pool-e3fe9fe1c13743478e09f5eeeb528b57.queue_time_us) (tablet-download-pool-f1e048df08cf4f7eaacb78d6592d79b1.queue_time_us, tablet-download-pool-f1e048df08cf4f7eaacb78d6592d79b1.queue_time_us) (tablet-download-pool-f1e048df08cf4f7eaacb78d6592d79b1.run_cpu_time_us, tablet-download-pool-f1e048df08cf4f7eaacb78d6592d79b1.run_cpu_time_us) (tablet-download-pool-f1e048df08cf4f7eaacb78d6592d79b1.run_wall_time_us, tablet-download-pool-f1e048df08cf4f7eaacb78d6592d79b1.run_wall_time_us) (tablet-download-pool-f67ca7eac95d4a7fbd28667bdde44a22.queue_time_us, tablet-download-pool-f67ca7eac95d4a7fbd28667bdde44a22.queue_time_us) (tablet-download-pool-f67ca7eac95d4a7fbd28667bdde44a22.run_cpu_time_us, tablet-download-pool-f67ca7eac95d4a7fbd28667bdde44a22.run_cpu_time_us) (tablet-download-pool-f67ca7eac95d4a7fbd28667bdde44a22.run_wall_time_us, tablet-download-pool-f67ca7eac95d4a7fbd28667bdde44a22.run_wall_time_us) (tablet-download-pool-fdb2924e91fb4a98aaa6bf977f682fdb.queue_time_us, tablet-download-pool-fdb2924e91fb4a98aaa6bf977f682fdb.queue_time_us) (tablet-download-pool-fdb2924e91fb4a98aaa6bf977f682fdb.run_cpu_time_us, tablet-download-pool-fdb2924e91fb4a98aaa6bf977f682fdb.run_cpu_time_us) (tablet-download-pool-fdb2924e91fb4a98aaa6bf977f682fdb.run_wall_time_us, tablet-download-pool-fdb2924e91fb4a98aaa6bf977f682fdb.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 14:01:12.043607 9740 ts_tablet_manager.cc:1875] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.051201 8633 raft_consensus.cc:2932] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:12.052017 5296 raft_consensus.cc:2932] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:12.055693 2146 raft_consensus.cc:2932] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:12.056968 2005 catalog_manager.cc:5484] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.059460 9612 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.060384 4901 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.069036 2000 catalog_manager.cc:5484] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:12.068974 5032 raft_consensus.cc:2932] T ce213e03db7f4f17913aad989c584607 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:12.074457 2407 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.080629 2406 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.081988 9727 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:12.082989 2277 consensus_queue.cc:237] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:12.087414 2146 raft_consensus.cc:1261] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.087538 8870 raft_consensus.cc:1050] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:12.088176 9271 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.089802 8870 consensus_queue.cc:237] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:12.091466 9733 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.095346 9611 ts_tablet_manager.cc:1398] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2: Time spent starting tablet: real 0.109s user 0.003s sys 0.000s
@ 0x7f465e448dcd google::LogMessage::Fail() at ??:0
I20240501 14:01:12.096889 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-fdb2924e91fb4a98aaa6bf977f682fdb received from {username='slave'} at 127.28.17.6:54429
I20240501 14:01:12.097030 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 6f81f803ec7746ccb502c59b7cbf8eb2-fdb2924e91fb4a98aaa6bf977f682fdb on tablet fdb2924e91fb4a98aaa6bf977f682fdb with peer 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:12.097693 5031 raft_consensus.cc:1203] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:12.093688 2146 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.109948 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 566acdef1d114b8aac9b55ccfa03da35 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.113502 5294 raft_consensus.cc:1261] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.114137 9733 consensus_queue.cc:1035] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.115936 5293 raft_consensus.cc:2932] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.116389 2004 catalog_manager.cc:5484] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:12.120220 2003 catalog_manager.cc:5484] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:12.121256 9735 ts_tablet_manager.cc:1888] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.121384 9735 log.cc:1196] T ead586b9a8024a218c55d2bb579a16de P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/ead586b9a8024a218c55d2bb579a16de
I20240501 14:01:12.122117 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet ead586b9a8024a218c55d2bb579a16de (table rebalancer_extra_table_42 [id=ad3b306c01b74211a105e2e7c38b719e]) successfully deleted
I20240501 14:01:12.135564 5013 tablet_service.cc:1514] Processing DeleteTablet for tablet 566acdef1d114b8aac9b55ccfa03da35 with delete_type TABLET_DATA_TOMBSTONED (TS 6f81f803ec7746ccb502c59b7cbf8eb2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58280
I20240501 14:01:12.147897 9740 ts_tablet_manager.cc:1888] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.148027 9740 log.cc:1196] T 0fac2baeae824f68ae2fe1c62e36d970 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/0fac2baeae824f68ae2fe1c62e36d970
@ 0x7f465e44cb93 google::LogMessage::SendToLog() at ??:0
I20240501 14:01:12.155889 9716 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:12.156255 9716 tablet_bootstrap.cc:492] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:12.156498 9716 ts_tablet_manager.cc:1367] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.140s user 0.012s sys 0.001s
I20240501 14:01:12.157215 9716 raft_consensus.cc:348] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:12.157459 9716 raft_consensus.cc:729] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:12.157649 9716 consensus_queue.cc:260] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:12.159054 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 0fac2baeae824f68ae2fe1c62e36d970 (table rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]) successfully deleted
I20240501 14:01:12.162320 9696 raft_consensus.cc:1050] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:01:12.162320 9716 ts_tablet_manager.cc:1398] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:01:12.163085 9696 consensus_queue.cc:237] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:12.167016 5161 consensus_queue.cc:237] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:12.167580 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-b911b0ff6ef446d8a6637edca57e570d received from {username='slave'} at 127.28.17.2:35917
I20240501 14:01:12.167728 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-b911b0ff6ef446d8a6637edca57e570d on tablet b911b0ff6ef446d8a6637edca57e570d with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:12.172205 2274 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.173596 9740 tablet_replica.cc:330] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: stopping tablet replica
I20240501 14:01:12.173884 9727 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:12.174193 9727 tablet_bootstrap.cc:492] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:12.174372 9727 ts_tablet_manager.cc:1367] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.008s
I20240501 14:01:12.175815 9727 raft_consensus.cc:348] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:12.176028 9727 raft_consensus.cc:729] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:12.176214 9727 consensus_queue.cc:260] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:12.176837 2407 raft_consensus.cc:1261] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.176949 9727 ts_tablet_manager.cc:1398] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:12.177596 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-beb5db06e4b74430b2bcbbecc82796e2 received from {username='slave'} at 127.28.17.4:35953
I20240501 14:01:12.177738 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-beb5db06e4b74430b2bcbbecc82796e2 on tablet beb5db06e4b74430b2bcbbecc82796e2 with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:12.178398 9740 raft_consensus.cc:2229] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.178550 9390 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
@ 0x7f465e4487cc google::LogMessage::Flush() at ??:0
@ 0x7f465e449f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:01:12.181166 9609 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.183182 9740 raft_consensus.cc:2258] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.185832 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ed332b3f7304bf69bc348d230ece128 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.187074 2277 raft_consensus.cc:1261] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.189869 5032 raft_consensus.cc:1261] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.190435 9515 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.190294 2000 catalog_manager.cc:5484] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:12.194471 2407 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.195911 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed332b3f7304bf69bc348d230ece128 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:12.196666 9768 tablet_replica.cc:330] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:12.196905 9768 raft_consensus.cc:2229] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.197176 9768 raft_consensus.cc:2258] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.198338 9768 ts_tablet_manager.cc:1875] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.198590 5294 raft_consensus.cc:1261] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.199571 8701 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.205144 9768 ts_tablet_manager.cc:1888] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.205282 9768 log.cc:1196] T 4ed332b3f7304bf69bc348d230ece128 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/4ed332b3f7304bf69bc348d230ece128
I20240501 14:01:12.206086 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 4ed332b3f7304bf69bc348d230ece128 (table rebalancer_extra_table_20 [id=dc6bf4405f334be39185f4fb02cdf5ae]) successfully deleted
I20240501 14:01:12.206753 9609 consensus_queue.cc:1035] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.208576 2274 raft_consensus.cc:2932] T 4ed332b3f7304bf69bc348d230ece128 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:12.211577 9434 raft_consensus.cc:1050] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:01:12.212354 9434 consensus_queue.cc:237] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:12.217267 4901 raft_consensus.cc:1261] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.219429 2538 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.220566 9763 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.223022 4901 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.223623 9527 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.224929 5161 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.226307 9434 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.228199 9740 ts_tablet_manager.cc:1875] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.228365 2538 consensus_queue.cc:237] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:12.232368 9515 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.233417 2274 raft_consensus.cc:1261] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.233971 9749 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.236543 9749 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.239243 2274 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.245234 9515 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.246870 5031 raft_consensus.cc:1261] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.247501 9434 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.250126 4901 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.251717 9465 consensus_queue.cc:1035] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.253407 4899 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.253595 5031 raft_consensus.cc:2932] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.259920 2005 catalog_manager.cc:5484] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.266088 2001 catalog_manager.cc:5484] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:12.267154 5294 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.271126 5031 raft_consensus.cc:1261] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.271690 9515 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.282718 2538 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.287788 2001 catalog_manager.cc:5484] T 9b91e5a14d82419cb295439f577d2786 P 6f81f803ec7746ccb502c59b7cbf8eb2 reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.292002 5031 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.292503 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b91e5a14d82419cb295439f577d2786 with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:12.298110 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b91e5a14d82419cb295439f577d2786 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.304816 2277 raft_consensus.cc:1261] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.306428 5162 raft_consensus.cc:1261] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.308313 8701 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.312331 2146 consensus_queue.cc:237] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:12.315841 5032 raft_consensus.cc:1261] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.316310 4901 raft_consensus.cc:1261] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.316370 8701 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.316835 9515 consensus_queue.cc:1035] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.319417 9515 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.319729 5031 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.322952 8701 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.337703 2537 raft_consensus.cc:1203] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:12.339258 5161 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.343868 4901 raft_consensus.cc:2932] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.352461 2005 catalog_manager.cc:5484] T 0e407965bc6047b185913b10dd0681fa P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.365751 8701 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.372888 5034 consensus_queue.cc:237] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:12.380703 9780 tablet_replica.cc:330] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:12.380919 9780 raft_consensus.cc:2229] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.381125 9780 raft_consensus.cc:2258] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.385967 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e407965bc6047b185913b10dd0681fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.385644 5161 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.386785 9780 ts_tablet_manager.cc:1875] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.390588 5033 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.396324 2006 catalog_manager.cc:5484] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.399744 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e407965bc6047b185913b10dd0681fa with delete_type TABLET_DATA_TOMBSTONED (TS 5a4b9a96e9734baa8a03d6056bfa934c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36154
I20240501 14:01:12.410465 2274 raft_consensus.cc:1203] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:12.410967 2277 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.415650 9749 raft_consensus.cc:1050] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:12.416529 9749 consensus_queue.cc:237] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:12.422621 9780 ts_tablet_manager.cc:1888] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.422775 9780 log.cc:1196] T 9b91e5a14d82419cb295439f577d2786 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/9b91e5a14d82419cb295439f577d2786
I20240501 14:01:12.430907 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 9b91e5a14d82419cb295439f577d2786 (table rebalancer_extra_table_28 [id=d1861840c97941dc8197bed58b21816f]) successfully deleted
I20240501 14:01:12.433951 2146 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:12.434660 8633 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:12.434942 5296 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:12.435565 8633 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:12.442905 4901 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:12.449648 9465 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:12.464021 4900 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.469599 9763 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.476614 5162 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 6f81f803ec7746ccb502c59b7cbf8eb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.476327 9527 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.477088 4900 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.478751 9763 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P 6f81f803ec7746ccb502c59b7cbf8eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240501 14:01:12.482795 5162 raft_consensus.cc:2932] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.485996 2001 catalog_manager.cc:5484] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) evicted. New cstate: current_term: 1 leader_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:12.487298 2146 consensus_queue.cc:237] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:12.490304 2277 raft_consensus.cc:1261] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.493366 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet f647f3f27faf435f81e076bfc2ef0795 with delete_type TABLET_DATA_TOMBSTONED (TS 745ac12076e84effa5bde1ecbd6a32ec not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44554
@ 0x7f4661189742 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:01:12.507014 9790 tablet_replica.cc:330] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:01:12.507525 8870 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.509435 5025 raft_consensus.cc:1261] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.509723 9798 tablet_replica.cc:330] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: stopping tablet replica
I20240501 14:01:12.509923 9798 raft_consensus.cc:2229] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.510157 9798 raft_consensus.cc:2258] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.516134 9798 ts_tablet_manager.cc:1875] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.518960 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f647f3f27faf435f81e076bfc2ef0795 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.526436 9390 raft_consensus.cc:1050] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 to VOTER
I20240501 14:01:12.527391 9390 consensus_queue.cc:237] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:12.534157 2407 raft_consensus.cc:1261] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.534610 9696 consensus_queue.cc:1035] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.534937 2277 raft_consensus.cc:1261] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.535631 9390 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.537500 8870 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.538626 9803 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.541503 5025 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.542721 5031 raft_consensus.cc:1261] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.543865 9390 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.550022 4900 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "741357494d3a4842935671afd1dbe677"
dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57954
I20240501 14:01:12.550316 4900 raft_consensus.cc:595] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:12.556200 9798 ts_tablet_manager.cc:1888] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.558403 9390 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.562230 2001 catalog_manager.cc:5484] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:12.562832 9798 log.cc:1196] T f647f3f27faf435f81e076bfc2ef0795 P 745ac12076e84effa5bde1ecbd6a32ec: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/wals/f647f3f27faf435f81e076bfc2ef0795
I20240501 14:01:12.563145 2406 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.507283 9790 raft_consensus.cc:2229] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.565666 9790 raft_consensus.cc:2258] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.567777 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a489f3c24c54297a0569dd560cf159e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.568512 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a489f3c24c54297a0569dd560cf159e with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:12.568732 9790 ts_tablet_manager.cc:1875] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.568962 9780 tablet_replica.cc:330] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:12.569170 9780 raft_consensus.cc:2229] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.569425 9780 raft_consensus.cc:2258] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.570664 1990 catalog_manager.cc:4830] TS 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): tablet f647f3f27faf435f81e076bfc2ef0795 (table rebalancer_extra_table_48 [id=5a4f7d80ee1547c2a8ccbe684a985094]) successfully deleted
I20240501 14:01:12.570853 9780 ts_tablet_manager.cc:1875] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.571251 2001 catalog_manager.cc:5484] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.573293 9740 ts_tablet_manager.cc:1888] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.573422 9740 log.cc:1196] T 566acdef1d114b8aac9b55ccfa03da35 P 6f81f803ec7746ccb502c59b7cbf8eb2: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/wals/566acdef1d114b8aac9b55ccfa03da35
I20240501 14:01:12.574127 1992 catalog_manager.cc:4830] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): tablet 566acdef1d114b8aac9b55ccfa03da35 (table rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]) successfully deleted
I20240501 14:01:12.584602 2274 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.585798 8633 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.592836 4901 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.598358 9780 ts_tablet_manager.cc:1888] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.598618 9780 log.cc:1196] T 0a489f3c24c54297a0569dd560cf159e P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/0a489f3c24c54297a0569dd560cf159e
I20240501 14:01:12.599686 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 0a489f3c24c54297a0569dd560cf159e (table rebalancer_extra_table_36 [id=b8fe210c34f149b1bc9fb4b477679d11]) successfully deleted
I20240501 14:01:12.605669 5296 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
@ 0x7f4661155f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 14:01:12.613421 2274 raft_consensus.cc:2932] T 0a489f3c24c54297a0569dd560cf159e P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.616495 2006 catalog_manager.cc:5484] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.621845 5031 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.635053 5161 consensus_queue.cc:237] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:12.637229 2274 raft_consensus.cc:1261] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.637322 5031 raft_consensus.cc:1261] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.637742 9390 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.638635 9803 consensus_queue.cc:1035] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.641772 9609 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.642439 2277 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
@ 0x7f466114fcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 14:01:12.655436 2146 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.656345 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fdb2924e91fb4a98aaa6bf977f682fdb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.657943 2001 catalog_manager.cc:5484] T fdb2924e91fb4a98aaa6bf977f682fdb P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.661751 2537 consensus_queue.cc:237] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:12.667117 9758 raft_consensus.cc:979] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: : Instructing follower a64f34785a7b44fa8070b0452a4e0ddd to start an election
I20240501 14:01:12.666798 5031 raft_consensus.cc:2932] T fdb2924e91fb4a98aaa6bf977f682fdb P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } }
I20240501 14:01:12.667254 9758 raft_consensus.cc:1067] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Signalling peer a64f34785a7b44fa8070b0452a4e0ddd to start an election
I20240501 14:01:12.668056 2407 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "741357494d3a4842935671afd1dbe677"
dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
from {username='slave'} at 127.28.17.5:49081
I20240501 14:01:12.668300 2407 raft_consensus.cc:482] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:12.668407 2407 raft_consensus.cc:3037] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:12.673519 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet fdb2924e91fb4a98aaa6bf977f682fdb with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51622
I20240501 14:01:12.681464 2407 raft_consensus.cc:504] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:12.682737 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741357494d3a4842935671afd1dbe677" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9"
I20240501 14:01:12.682978 4900 raft_consensus.cc:3032] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:12.683058 4900 raft_consensus.cc:729] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:12.683400 4900 consensus_queue.cc:260] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:12.683809 4900 raft_consensus.cc:3037] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:12.687150 4900 raft_consensus.cc:2454] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 2.
I20240501 14:01:12.688617 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741357494d3a4842935671afd1dbe677" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:12.688861 5161 raft_consensus.cc:3037] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:12.695865 5161 raft_consensus.cc:2454] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 2.
I20240501 14:01:12.696365 2341 leader_election.cc:302] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:12.696424 5293 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:12.696657 9770 raft_consensus.cc:2781] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:12.696761 9770 raft_consensus.cc:686] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:01:12.697008 9813 tablet_replica.cc:330] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:01:12.697211 9813 raft_consensus.cc:2229] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.697074 9770 consensus_queue.cc:237] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:12.697455 9813 raft_consensus.cc:2258] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.698184 8633 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:12.702116 2407 leader_election.cc:288] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:12.701858 5293 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.703305 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741357494d3a4842935671afd1dbe677" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:01:12.703488 2146 raft_consensus.cc:3037] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:12.708545 2146 raft_consensus.cc:2454] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 2.
I20240501 14:01:12.708545 2000 catalog_manager.cc:5484] T 1cb2dab6f4684bb18bddc9edba1d6517 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.709421 9813 ts_tablet_manager.cc:1875] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.709885 9790 ts_tablet_manager.cc:1888] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.710002 9790 log.cc:1196] T 0e407965bc6047b185913b10dd0681fa P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/0e407965bc6047b185913b10dd0681fa
I20240501 14:01:12.710644 9465 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.712846 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cb2dab6f4684bb18bddc9edba1d6517 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48706
I20240501 14:01:12.714251 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1cb2dab6f4684bb18bddc9edba1d6517 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.723431 2000 catalog_manager.cc:5484] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 1 to 2, leader changed from f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 2 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:12.725600 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 0e407965bc6047b185913b10dd0681fa (table rebalancer_extra_table_54 [id=1ce7796dfef044868b9485fac7e54cfd]) successfully deleted
@ 0x7f46705a28de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:01:12.732364 4900 raft_consensus.cc:1261] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:12.732898 9808 consensus_queue.cc:1035] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:12.734364 9813 ts_tablet_manager.cc:1888] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.734516 9813 log.cc:1196] T fdb2924e91fb4a98aaa6bf977f682fdb P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/fdb2924e91fb4a98aaa6bf977f682fdb
I20240501 14:01:12.735255 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet fdb2924e91fb4a98aaa6bf977f682fdb (table rebalancer_extra_table_60 [id=38aca7990f7b44f493751dc289a92891]) successfully deleted
I20240501 14:01:12.735087 4900 raft_consensus.cc:2932] T 1cb2dab6f4684bb18bddc9edba1d6517 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:12.748637 9816 tablet_replica.cc:330] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:01:12.748860 9816 raft_consensus.cc:2229] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.749140 9816 raft_consensus.cc:2258] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.754901 9816 ts_tablet_manager.cc:1875] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.768981 9778 raft_consensus.cc:1050] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:12.769834 9778 consensus_queue.cc:237] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
@ 0x7f46705badaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:01:12.795858 9816 ts_tablet_manager.cc:1888] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:12.795862 2537 raft_consensus.cc:1261] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.795992 9816 log.cc:1196] T 1cb2dab6f4684bb18bddc9edba1d6517 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/1cb2dab6f4684bb18bddc9edba1d6517
I20240501 14:01:12.796581 9779 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.796720 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 1cb2dab6f4684bb18bddc9edba1d6517 (table rebalance_test_table_17 [id=bca565af768e459db2524d9f31ad6a28]) successfully deleted
I20240501 14:01:12.799968 4900 raft_consensus.cc:1261] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.801448 9776 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.805240 5161 raft_consensus.cc:1261] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.806655 9776 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.847833 9779 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
W20240501 14:01:12.850864 9822 log.cc:924] Time spent T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240501 14:01:12.856894 5162 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
@ 0x7f46706e5480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:01:12.871801 2001 catalog_manager.cc:5484] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
@ 0x7f467071da8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 14:01:12.890552 2537 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:12.897512 5293 consensus_queue.cc:237] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:12.899739 9784 raft_consensus.cc:1050] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:12.900605 2538 raft_consensus.cc:1261] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.900563 9784 consensus_queue.cc:237] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:12.903481 9782 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.893991 4901 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:12.912708 9782 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:12.917336 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet beb5db06e4b74430b2bcbbecc82796e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.918315 2537 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:12.924074 2277 raft_consensus.cc:1261] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.924190 5031 raft_consensus.cc:1261] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.924631 9778 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.924938 5162 raft_consensus.cc:1261] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:12.925420 9778 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.929708 2006 catalog_manager.cc:5484] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f4670700981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 14:01:12.931089 5161 raft_consensus.cc:1261] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.932354 9784 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:12.932967 9784 consensus_queue.cc:1035] T beb5db06e4b74430b2bcbbecc82796e2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.934646 5162 raft_consensus.cc:2932] T beb5db06e4b74430b2bcbbecc82796e2 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:12.938524 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet beb5db06e4b74430b2bcbbecc82796e2 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:12.947367 9778 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.951350 2277 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.952250 9837 tablet_replica.cc:330] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
@ 0x7f467070070e std::_Function_handler<>::_M_invoke() at ??:0
I20240501 14:01:12.952450 9837 raft_consensus.cc:2229] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.952765 9837 raft_consensus.cc:2258] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.953624 9837 ts_tablet_manager.cc:1875] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.950899 2537 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce213e03db7f4f17913aad989c584607"
dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40676
I20240501 14:01:12.954603 2537 raft_consensus.cc:595] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:12.955204 2001 catalog_manager.cc:5484] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:12.957422 5031 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.961699 9837 ts_tablet_manager.cc:1888] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.961828 9837 log.cc:1196] T beb5db06e4b74430b2bcbbecc82796e2 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/beb5db06e4b74430b2bcbbecc82796e2
I20240501 14:01:12.963946 5293 consensus_queue.cc:237] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:01:12.965881 2277 raft_consensus.cc:1261] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.967404 9779 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.968029 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet beb5db06e4b74430b2bcbbecc82796e2 (table rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]) successfully deleted
I20240501 14:01:12.969895 9778 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.970577 2277 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:12.974071 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b911b0ff6ef446d8a6637edca57e570d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:12.978893 2001 catalog_manager.cc:5484] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:12.983326 5026 raft_consensus.cc:1261] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:12.983705 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet b911b0ff6ef446d8a6637edca57e570d with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:12.984067 9780 tablet_replica.cc:330] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:12.984301 9780 raft_consensus.cc:2229] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:12.984277 9778 consensus_queue.cc:1035] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:12.984524 9780 pending_rounds.cc:62] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Trying to abort 1 pending ops.
I20240501 14:01:12.984635 9780 pending_rounds.cc:68] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022887210598481920 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b911b0ff6ef446d8a6637edca57e570d" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } } }
I20240501 14:01:12.985040 9780 raft_consensus.cc:2894] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:01:12.985172 9780 raft_consensus.cc:2866] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:01:12.985282 9780 raft_consensus.cc:2258] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:12.986893 9780 ts_tablet_manager.cc:1875] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:12.988279 5026 raft_consensus.cc:2932] T b911b0ff6ef446d8a6637edca57e570d P 6f81f803ec7746ccb502c59b7cbf8eb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
@ 0x7f4661159b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 14:01:12.996428 9780 ts_tablet_manager.cc:1888] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:12.996580 9780 log.cc:1196] T b911b0ff6ef446d8a6637edca57e570d P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/b911b0ff6ef446d8a6637edca57e570d
I20240501 14:01:12.997334 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet b911b0ff6ef446d8a6637edca57e570d (table rebalancer_extra_table_30 [id=b565235bf59f49af828ab4731a784d68]) successfully deleted
@ 0x7f466115d19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f466115d019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f466113801e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f46644bd6db start_thread at ??:0
@ 0x7f465d2fb71f clone at ??:0
I20240501 14:01:13.137378 9465 raft_consensus.cc:979] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec: : Instructing follower d1a414d381f44829a2510cb76d481887 to start an election
I20240501 14:01:13.137512 9465 raft_consensus.cc:1067] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Signalling peer d1a414d381f44829a2510cb76d481887 to start an election
I20240501 14:01:13.139308 2145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce213e03db7f4f17913aad989c584607"
dest_uuid: "d1a414d381f44829a2510cb76d481887"
from {username='slave'} at 127.28.17.4:38549
I20240501 14:01:13.139480 2145 raft_consensus.cc:482] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:13.139549 2145 raft_consensus.cc:3037] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:13.141955 2145 raft_consensus.cc:504] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:13.147927 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:13.148123 2537 raft_consensus.cc:3032] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:13.148029 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:13.148171 2537 raft_consensus.cc:729] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:01:13.148247 5291 raft_consensus.cc:3037] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:13.148407 2537 consensus_queue.cc:260] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:13.148705 2537 raft_consensus.cc:3037] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:13.151203 2537 raft_consensus.cc:2454] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:13.151257 5291 raft_consensus.cc:2454] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:13.151922 2145 leader_election.cc:288] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:13.152456 2080 leader_election.cc:302] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:13.152721 9696 raft_consensus.cc:2781] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:13.152837 9696 raft_consensus.cc:686] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:13.153101 9696 consensus_queue.cc:237] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:13.158483 4901 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:13.159148 9814 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:13.178594 2001 catalog_manager.cc:5484] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 1 to 2, leader changed from 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:13.223629 2145 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:13.224884 9814 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:13.234416 5161 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:13.235456 9770 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:13.492473 2213 connection.cc:535] client connection to 127.28.17.6:46037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.492517 5230 connection.cc:535] client connection to 127.28.17.6:46037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.492738 2213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.492755 5230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.513831 2343 connection.cc:535] client connection to 127.28.17.6:46037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.514158 2343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.528345 2473 connection.cc:535] client connection to 127.28.17.6:46037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.528401 2082 connection.cc:535] client connection to 127.28.17.6:46037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.528592 2473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.528668 2082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.538453 4836 connection.cc:535] client connection to 127.28.17.6:46037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.538775 4836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.543534 5099 connection.cc:535] client connection to 127.28.17.6:46037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.543843 5099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:13.544615 9909 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:13.511717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.6:46037 (local address 127.28.17.2:50223)
0501 14:01:13.518173 (+ 6456us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:13.518195 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0501 14:01:13.518345 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:01:13.543693 (+ 25348us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":6336,"thread_start_us":6710,"threads_started":1}
W20240501 14:01:13.544615 9915 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:13.541949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.6:46037 (local address 127.28.17.4:36383)
0501 14:01:13.542410 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:13.542430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0501 14:01:13.542601 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:01:13.544030 (+ 1429us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
W20240501 14:01:13.544775 2213 consensus_peers.cc:485] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.544813 2473 consensus_peers.cc:485] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.544871 2213 consensus_peers.cc:485] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.545034 2213 consensus_peers.cc:485] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.545125 2213 consensus_peers.cc:485] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.545181 2213 consensus_peers.cc:485] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.545290 2213 consensus_peers.cc:485] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.545349 2473 consensus_peers.cc:485] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.545430 2473 consensus_peers.cc:485] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.545779 2473 consensus_peers.cc:485] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.546300 2213 consensus_peers.cc:485] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.546746 2473 consensus_peers.cc:485] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.552011 9911 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:13.517296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.6:46037 (local address 127.28.17.3:50911)
0501 14:01:13.550842 (+ 33546us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:13.551005 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33391,"threads_started":1}
W20240501 14:01:13.553550 2343 consensus_peers.cc:485] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.556479 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:13.498934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.6:46037 (local address 127.28.17.8:53605)
0501 14:01:13.555222 (+ 56288us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:13.555395 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":56139,"threads_started":1}
I20240501 14:01:13.558256 5291 raft_consensus.cc:1261] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240501 14:01:13.560237 2343 consensus_peers.cc:485] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.560369 2343 consensus_peers.cc:485] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.560431 2343 consensus_peers.cc:485] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.560499 2343 consensus_peers.cc:485] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.560554 2343 consensus_peers.cc:485] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.560993 5230 consensus_peers.cc:485] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.561156 5230 consensus_peers.cc:485] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.569857 2537 raft_consensus.cc:1261] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240501 14:01:13.570464 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:13.571708 5230 consensus_peers.cc:485] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.572625 4836 consensus_peers.cc:485] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.571352 9696 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:13.573500 2082 leader_election.cc:334] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:13.573869 4836 consensus_peers.cc:485] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.574296 2082 consensus_peers.cc:485] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.574549 4836 consensus_peers.cc:485] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.574733 4836 consensus_peers.cc:485] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.574832 4836 consensus_peers.cc:485] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.575208 2082 consensus_peers.cc:485] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.575309 2082 consensus_peers.cc:485] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.575623 9696 consensus_queue.cc:1035] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:13.578811 5099 consensus_peers.cc:485] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.580054 5099 consensus_peers.cc:485] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.580195 5099 consensus_peers.cc:485] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.588532 2082 consensus_peers.cc:485] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.655967 2082 consensus_peers.cc:485] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.666812 5161 consensus_queue.cc:237] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
W20240501 14:01:13.669111 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:13.678576 4836 consensus_peers.cc:485] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.678426 5291 raft_consensus.cc:1261] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.679427 9942 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:13.679746 5097 consensus_peers.cc:485] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52eb73b91ed441199eade33e66b84b39. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.680464 2537 raft_consensus.cc:1261] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.681406 9942 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:13.694253 9939 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:13.696415 2537 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
W20240501 14:01:13.704746 2082 consensus_peers.cc:485] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.706897 2000 catalog_manager.cc:5484] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:13.713055 5296 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:13.750792 5162 consensus_queue.cc:237] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:13.753293 4901 raft_consensus.cc:1261] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.753494 2146 raft_consensus.cc:1261] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.753899 9609 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:13.754201 9942 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:13.754601 5099 consensus_peers.cc:485] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ff679e55a5446099efbdbcd1109f245. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.756721 9942 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:13.761972 4901 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:13.772567 2005 catalog_manager.cc:5484] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:13.772924 2146 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:13.775162 2407 consensus_queue.cc:237] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:13.778856 5162 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:13.778751 2146 raft_consensus.cc:1261] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:13.781471 9814 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:13.781843 9872 consensus_queue.cc:1035] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:13.784200 9872 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:13.784832 2146 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:13.787714 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 741357494d3a4842935671afd1dbe677 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:13.788070 5162 raft_consensus.cc:2932] T 741357494d3a4842935671afd1dbe677 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:13.792258 2004 catalog_manager.cc:5484] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 3 to 5, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 2 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:13.801918 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 741357494d3a4842935671afd1dbe677 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:13.802721 9966 tablet_replica.cc:330] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:13.803009 9966 raft_consensus.cc:2229] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:13.803231 9966 raft_consensus.cc:2258] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:13.804539 9966 ts_tablet_manager.cc:1875] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:13.810848 9966 ts_tablet_manager.cc:1888] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 14:01:13.810994 9966 log.cc:1196] T 741357494d3a4842935671afd1dbe677 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/741357494d3a4842935671afd1dbe677
I20240501 14:01:13.811749 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 741357494d3a4842935671afd1dbe677 (table rebalancer_extra_table_46 [id=6ee7f160bc69454cb2a2a0c7d0b17a04]) successfully deleted
I20240501 14:01:13.816972 2274 consensus_queue.cc:237] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
W20240501 14:01:13.820374 2213 consensus_peers.cc:485] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:13.820613 2211 consensus_peers.cc:485] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c4312eec255417580288818bd559099. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.821470 5296 raft_consensus.cc:1261] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.822280 9945 consensus_queue.cc:1035] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:13.825264 9944 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:13.825781 5296 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:13.832619 2005 catalog_manager.cc:5484] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:13.867640 5161 consensus_queue.cc:237] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:13.875306 2407 raft_consensus.cc:1261] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:13.875463 5097 consensus_peers.cc:485] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccec6fa18f844ace900db8bfe91738a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.876523 9609 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:13.880048 5291 raft_consensus.cc:1261] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:13.881094 9609 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:13.885250 9942 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:13.885849 2407 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:13.888653 5291 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:13.893124 2001 catalog_manager.cc:5484] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:13.918175 2407 consensus_queue.cc:237] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:13.928067 5161 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:13.928670 9872 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:01:13.930419 2343 consensus_peers.cc:485] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9d8d993ce834c1db4dd4418f7a85780. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.932125 9814 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:13.932752 2277 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:13.934571 9770 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:13.935808 5161 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:13.944159 2001 catalog_manager.cc:5484] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:13.965582 2277 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:13.977946 9872 raft_consensus.cc:482] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:13.978140 9872 raft_consensus.cc:504] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:13.978852 9872 leader_election.cc:288] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:13.979714 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b11568c2d7420fa06f903e52723583" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:13.980113 2537 consensus_queue.cc:237] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:13.983804 4899 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:01:13.983908 5161 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:01:13.984375 2471 consensus_peers.cc:485] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb1fa60c64034b7d8c7b4879823b0d83. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:13.984509 9812 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:13.984795 9465 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:01:13.985073 2343 leader_election.cc:334] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:13.985206 2343 leader_election.cc:302] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:13.985741 9872 raft_consensus.cc:2726] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:13.989131 9465 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:13.989881 5161 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:13.997494 2001 catalog_manager.cc:5484] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.010413 4899 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } } }
I20240501 14:01:14.027078 4901 consensus_queue.cc:237] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:14.029949 2146 raft_consensus.cc:1261] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:14.030591 9875 consensus_queue.cc:1035] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:14.031155 4834 consensus_peers.cc:485] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afc49574262e4819b6c86025c43c772a. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:14.031569 4836 consensus_peers.cc:485] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.033403 9758 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:14.034188 2146 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:14.040097 2001 catalog_manager.cc:5484] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:14.052650 9465 raft_consensus.cc:482] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.052825 9465 raft_consensus.cc:504] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:14.053496 9465 leader_election.cc:288] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.054286 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c685294d3fca4677bc1b1cf355dac3ce" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:01:14.054929 2473 leader_election.cc:334] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.054986 9942 raft_consensus.cc:482] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.055073 2473 leader_election.cc:302] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:14.055169 9942 raft_consensus.cc:504] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.055322 9465 raft_consensus.cc:2726] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.055799 9942 leader_election.cc:288] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:14.056334 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9790fa13fe4cb286f0fb7f730da336" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:01:14.057610 5099 leader_election.cc:334] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.057766 5099 leader_election.cc:302] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:14.058514 9758 raft_consensus.cc:482] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.058717 9758 raft_consensus.cc:504] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:14.059401 9758 leader_election.cc:288] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:14.060109 9942 raft_consensus.cc:2726] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.061043 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c685294d3fca4677bc1b1cf355dac3ce" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:14.061242 2537 raft_consensus.cc:2454] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
W20240501 14:01:14.061767 4836 leader_election.cc:334] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.062673 4834 leader_election.cc:302] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.062938 9758 raft_consensus.cc:2781] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.063040 9758 raft_consensus.cc:482] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.063148 9758 raft_consensus.cc:3037] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.065454 9758 raft_consensus.cc:504] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:14.065724 2277 consensus_queue.cc:237] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:14.066041 9758 leader_election.cc:288] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:14.066469 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c685294d3fca4677bc1b1cf355dac3ce" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:14.066674 2537 raft_consensus.cc:3037] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:14.067427 4836 leader_election.cc:334] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.068666 5161 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:14.068682 2211 consensus_peers.cc:485] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efbd148f37bb46f0a4bf0fc562fff597. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.069065 2537 raft_consensus.cc:2454] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:14.069140 2538 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:14.069756 4834 leader_election.cc:302] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.070158 9758 raft_consensus.cc:2781] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.070142 9733 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.070334 9758 raft_consensus.cc:686] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:14.070627 9758 consensus_queue.cc:237] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:14.070683 9733 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.075198 9943 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:14.076252 5161 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:14.078876 9770 raft_consensus.cc:482] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.079115 9770 raft_consensus.cc:504] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:14.079753 9770 leader_election.cc:288] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.082651 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926fd4668e8644969365d57749be3e1f" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:01:14.086279 2343 leader_election.cc:334] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.086421 2343 leader_election.cc:302] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:14.087095 9770 raft_consensus.cc:2726] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.086361 2005 catalog_manager.cc:5484] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.088670 2001 catalog_manager.cc:5484] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.096954 9770 raft_consensus.cc:482] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.097146 9770 raft_consensus.cc:504] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.097781 9770 leader_election.cc:288] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:14.099179 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0724625a6a8e4ce38e8087472b856c5c" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
W20240501 14:01:14.100140 2343 leader_election.cc:334] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.100306 2343 leader_election.cc:302] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.100585 9770 raft_consensus.cc:2726] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.102644 2538 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:14.115932 5161 consensus_queue.cc:237] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
W20240501 14:01:14.118742 5097 consensus_peers.cc:485] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36f03024767442ac911a8a44f7ecdceb. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.118775 5296 raft_consensus.cc:1261] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:14.118908 5099 consensus_peers.cc:485] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.119256 9940 consensus_queue.cc:1035] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.121701 9609 raft_consensus.cc:2932] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:14.122429 5296 raft_consensus.cc:2932] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:14.128139 2000 catalog_manager.cc:5484] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.132462 9696 raft_consensus.cc:482] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.132658 9696 raft_consensus.cc:504] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.133554 9696 leader_election.cc:288] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.134370 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5086890bd254edebdf12812ff5c3728" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
W20240501 14:01:14.135295 2082 leader_election.cc:334] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.135506 2080 leader_election.cc:302] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:14.135768 9696 raft_consensus.cc:2726] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.152289 10031 ts_tablet_manager.cc:902] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:14.154985 10031 tablet_copy_client.cc:308] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:14.155665 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 52eb73b91ed441199eade33e66b84b39 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:40205)
I20240501 14:01:14.155858 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 52eb73b91ed441199eade33e66b84b39 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:40205: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-52eb73b91ed441199eade33e66b84b39
I20240501 14:01:14.158421 5184 tablet_copy_source_session.cc:205] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.160562 10031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52eb73b91ed441199eade33e66b84b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.165480 9609 raft_consensus.cc:482] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.165652 9609 raft_consensus.cc:504] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:14.166373 9609 leader_election.cc:288] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.166868 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913ae5fda9e842e1b3a6d7dc418a5bfd" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:01:14.167575 10031 tablet_copy_client.cc:783] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.167790 10031 tablet_copy_client.cc:647] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
W20240501 14:01:14.169124 5099 leader_election.cc:334] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.169267 5099 leader_election.cc:302] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887
I20240501 14:01:14.169507 9609 raft_consensus.cc:2726] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.170154 10031 tablet_copy_client.cc:523] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.173604 10031 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:14.178562 9770 raft_consensus.cc:482] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.178719 9770 raft_consensus.cc:504] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:14.179327 9770 leader_election.cc:288] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:14.181669 2343 leader_election.cc:334] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.181720 10031 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.181953 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf051cecef424db5accfce66623346b5" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:01:14.182068 10031 tablet_bootstrap.cc:492] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:14.182336 10031 ts_tablet_manager.cc:1367] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240501 14:01:14.182615 2341 leader_election.cc:302] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887
I20240501 14:01:14.182950 9770 raft_consensus.cc:2726] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.182999 10031 raft_consensus.cc:348] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:14.183192 10031 raft_consensus.cc:729] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:14.183355 10031 consensus_queue.cc:260] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:14.183835 10031 ts_tablet_manager.cc:1398] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:01:14.184845 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-52eb73b91ed441199eade33e66b84b39 received from {username='slave'} at 127.28.17.2:40205
I20240501 14:01:14.184971 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-52eb73b91ed441199eade33e66b84b39 on tablet 52eb73b91ed441199eade33e66b84b39 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:14.195367 9778 raft_consensus.cc:482] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.195601 9778 raft_consensus.cc:504] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.196199 9778 leader_election.cc:288] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
W20240501 14:01:14.197954 5230 leader_election.cc:334] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.200852 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0724625a6a8e4ce38e8087472b856c5c" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:14.201102 2406 raft_consensus.cc:2454] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:01:14.202375 5228 leader_election.cc:302] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.202729 9778 raft_consensus.cc:2781] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.202859 9778 raft_consensus.cc:482] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.202963 9778 raft_consensus.cc:3037] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.205667 9778 raft_consensus.cc:504] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.207100 9778 leader_election.cc:288] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.208137 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0724625a6a8e4ce38e8087472b856c5c" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:14.208351 2406 raft_consensus.cc:3037] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:14.208802 5230 leader_election.cc:334] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.211097 2406 raft_consensus.cc:2454] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 2.
I20240501 14:01:14.212044 5228 leader_election.cc:302] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.212380 9778 raft_consensus.cc:2781] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.212561 9778 raft_consensus.cc:686] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:01:14.212807 9778 consensus_queue.cc:237] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.217691 2004 catalog_manager.cc:5484] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 2 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } }
W20240501 14:01:14.233494 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.320200 9778 raft_consensus.cc:482] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.320421 9778 raft_consensus.cc:504] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.321156 9778 leader_election.cc:288] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.321556 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147c5f61a574c7d8a334b5f67ce732d" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
W20240501 14:01:14.322527 5230 leader_election.cc:334] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.322680 5230 leader_election.cc:302] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:14.322927 9778 raft_consensus.cc:2726] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.327826 10115 ts_tablet_manager.cc:902] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:14.330423 10115 tablet_copy_client.cc:308] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:14.331077 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 0ff679e55a5446099efbdbcd1109f245 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:50045)
I20240501 14:01:14.331223 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 0ff679e55a5446099efbdbcd1109f245 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:50045: session id = 5a4b9a96e9734baa8a03d6056bfa934c-0ff679e55a5446099efbdbcd1109f245
I20240501 14:01:14.334216 5184 tablet_copy_source_session.cc:205] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.337175 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff679e55a5446099efbdbcd1109f245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.344245 10115 tablet_copy_client.cc:783] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.344573 10115 tablet_copy_client.cc:647] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.346277 10115 tablet_copy_client.cc:523] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.350328 10115 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:14.357471 10115 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.358024 10115 tablet_bootstrap.cc:492] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:14.358321 10115 ts_tablet_manager.cc:1367] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:01:14.359148 10115 raft_consensus.cc:348] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:14.359400 10115 raft_consensus.cc:729] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:01:14.359640 10115 consensus_queue.cc:260] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:14.360267 10115 ts_tablet_manager.cc:1398] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:14.362895 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-0ff679e55a5446099efbdbcd1109f245 received from {username='slave'} at 127.28.17.8:50045
I20240501 14:01:14.363034 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-0ff679e55a5446099efbdbcd1109f245 on tablet 0ff679e55a5446099efbdbcd1109f245 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:01:14.364984 9770 raft_consensus.cc:482] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.365137 9770 raft_consensus.cc:504] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.365799 9770 leader_election.cc:288] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:14.367481 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea342bfcde3b4daa8dcc39126fd42cdc" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:01:14.369105 2343 leader_election.cc:334] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.369849 2342 leader_election.cc:302] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:14.370141 9770 raft_consensus.cc:2726] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.375381 9942 raft_consensus.cc:482] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.375627 9942 raft_consensus.cc:504] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.376363 9942 leader_election.cc:288] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:14.377132 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:01:14.378557 5099 leader_election.cc:334] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.378688 5099 leader_election.cc:302] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:14.378989 9733 raft_consensus.cc:482] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.379138 9733 raft_consensus.cc:504] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:14.379755 9733 leader_election.cc:288] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.380824 9942 raft_consensus.cc:2726] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:14.383623 2213 leader_election.cc:334] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.383669 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:14.384313 9465 raft_consensus.cc:482] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.384454 9465 raft_consensus.cc:504] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.384680 2211 leader_election.cc:302] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:14.384804 10115 ts_tablet_manager.cc:902] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.385035 9465 leader_election.cc:288] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:14.385249 9733 raft_consensus.cc:2726] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.385406 9733 raft_consensus.cc:482] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.385502 9733 raft_consensus.cc:504] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.386063 9733 leader_election.cc:288] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.386271 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147c5f61a574c7d8a334b5f67ce732d" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:14.386529 5296 raft_consensus.cc:2454] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:14.386845 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b11568c2d7420fa06f903e52723583" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:14.387073 2406 raft_consensus.cc:2454] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 1.
W20240501 14:01:14.387248 2473 leader_election.cc:334] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.387486 2473 leader_election.cc:302] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.387864 9465 raft_consensus.cc:2781] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.387935 9465 raft_consensus.cc:482] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.388007 9465 raft_consensus.cc:3037] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.388417 2211 leader_election.cc:302] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:14.389693 9733 raft_consensus.cc:2781] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.389775 9733 raft_consensus.cc:482] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.389848 9733 raft_consensus.cc:3037] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.390872 9465 raft_consensus.cc:504] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.391489 9465 leader_election.cc:288] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
W20240501 14:01:14.391893 2213 leader_election.cc:334] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.393103 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147c5f61a574c7d8a334b5f67ce732d" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:14.393328 5296 raft_consensus.cc:3037] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.393612 9733 raft_consensus.cc:504] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
W20240501 14:01:14.393996 2473 leader_election.cc:334] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.395951 5296 raft_consensus.cc:2454] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 2.
I20240501 14:01:14.396422 2473 leader_election.cc:302] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.396849 9808 raft_consensus.cc:2781] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.396965 9808 raft_consensus.cc:686] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [term 2 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:01:14.397229 9808 consensus_queue.cc:237] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:14.402135 2004 catalog_manager.cc:5484] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 2 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.402781 9770 raft_consensus.cc:482] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.402940 9770 raft_consensus.cc:504] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.403611 9770 leader_election.cc:288] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:14.403963 2213 leader_election.cc:334] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.404433 9733 leader_election.cc:288] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.404724 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5086890bd254edebdf12812ff5c3728" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:01:14.404980 2146 raft_consensus.cc:2454] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:01:14.405836 2341 leader_election.cc:302] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters:
I20240501 14:01:14.406225 10004 raft_consensus.cc:482] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.406399 10004 raft_consensus.cc:504] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.407132 10004 leader_election.cc:288] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:14.407498 9770 raft_consensus.cc:2781] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.407572 9770 raft_consensus.cc:482] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.407634 9770 raft_consensus.cc:3037] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.408371 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9790fa13fe4cb286f0fb7f730da336" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:14.408597 5162 raft_consensus.cc:2454] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:14.409338 10115 tablet_copy_client.cc:308] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:01:14.410076 4834 leader_election.cc:302] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:14.410995 10004 raft_consensus.cc:2781] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.411127 10004 raft_consensus.cc:482] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.411216 10004 raft_consensus.cc:3037] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.413067 10004 raft_consensus.cc:504] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.413600 10004 leader_election.cc:288] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
W20240501 14:01:14.414597 4836 leader_election.cc:334] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.414639 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9790fa13fe4cb286f0fb7f730da336" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:14.414853 5162 raft_consensus.cc:3037] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:14.415536 4836 leader_election.cc:334] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.417021 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b11568c2d7420fa06f903e52723583" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:14.417229 2406 raft_consensus.cc:3037] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.417418 5162 raft_consensus.cc:2454] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
W20240501 14:01:14.417711 2343 leader_election.cc:334] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.417821 4834 leader_election.cc:302] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.418398 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet b9d8d993ce834c1db4dd4418f7a85780 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:53689)
I20240501 14:01:14.418532 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet b9d8d993ce834c1db4dd4418f7a85780 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:53689: session id = 5a4b9a96e9734baa8a03d6056bfa934c-b9d8d993ce834c1db4dd4418f7a85780
I20240501 14:01:14.419384 9974 raft_consensus.cc:2781] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.419534 9974 raft_consensus.cc:686] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:14.419832 9974 consensus_queue.cc:237] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.422617 2428 tablet_copy_source_session.cc:205] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.422457 9770 raft_consensus.cc:504] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.423449 9770 leader_election.cc:288] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Requested vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.424048 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5086890bd254edebdf12812ff5c3728" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887"
I20240501 14:01:14.424302 2146 raft_consensus.cc:3037] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.427464 2146 raft_consensus.cc:2454] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 2.
I20240501 14:01:14.427280 2004 catalog_manager.cc:5484] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:14.427683 9974 raft_consensus.cc:482] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.427865 9974 raft_consensus.cc:504] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:14.429037 9974 leader_election.cc:288] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.429136 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926fd4668e8644969365d57749be3e1f" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:14.429378 2407 raft_consensus.cc:2454] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:14.429888 4834 leader_election.cc:302] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:14.430219 9974 raft_consensus.cc:2781] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.430318 9974 raft_consensus.cc:482] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.430435 9974 raft_consensus.cc:3037] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.434103 9974 raft_consensus.cc:504] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:14.434448 2277 consensus_queue.cc:237] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
W20240501 14:01:14.435163 2343 leader_election.cc:334] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:14.435891 4836 leader_election.cc:334] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.436203 9814 raft_consensus.cc:482] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.436331 9814 raft_consensus.cc:504] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:14.436882 9814 leader_election.cc:288] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.437247 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:14.437424 2274 raft_consensus.cc:2454] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:01:14.437824 2341 leader_election.cc:302] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:14.438057 9770 raft_consensus.cc:2781] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.438153 9770 raft_consensus.cc:482] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.438241 9770 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.423640 2406 raft_consensus.cc:2454] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 2.
I20240501 14:01:14.439306 2341 leader_election.cc:302] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.439666 9974 leader_election.cc:288] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.439837 9814 raft_consensus.cc:2781] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.439980 9814 raft_consensus.cc:686] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:01:14.440143 2211 leader_election.cc:302] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.440258 9814 consensus_queue.cc:237] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.440532 9733 raft_consensus.cc:2781] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.440644 9733 raft_consensus.cc:686] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:01:14.441015 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d8d993ce834c1db4dd4418f7a85780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.440876 9733 consensus_queue.cc:237] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
W20240501 14:01:14.443240 4836 leader_election.cc:334] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.443410 2146 raft_consensus.cc:1261] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:14.443943 9945 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.444161 5161 raft_consensus.cc:1261] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:14.445147 2212 consensus_peers.cc:485] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fe1b54933ad446082b1d2840b2c6e49. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.446516 9945 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.449944 10115 tablet_copy_client.cc:783] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.450258 10115 tablet_copy_client.cc:647] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.459044 2004 catalog_manager.cc:5484] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 2 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.467367 2538 raft_consensus.cc:1261] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:14.467986 9974 consensus_queue.cc:1035] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:14.442443 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926fd4668e8644969365d57749be3e1f" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:14.481151 10140 ts_tablet_manager.cc:902] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:14.481418 2406 raft_consensus.cc:3037] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.483475 9733 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:14.484606 2406 raft_consensus.cc:2454] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:14.488783 9770 raft_consensus.cc:504] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:14.489424 9770 leader_election.cc:288] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Requested vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.490195 9974 raft_consensus.cc:482] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.490358 9974 raft_consensus.cc:504] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.491044 9974 leader_election.cc:288] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.493243 2004 catalog_manager.cc:5484] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.494988 5162 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
W20240501 14:01:14.502758 4836 leader_election.cc:334] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.503930 9696 raft_consensus.cc:482] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.504097 9696 raft_consensus.cc:504] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:14.504738 9696 leader_election.cc:288] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:14.506633 2082 leader_election.cc:334] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.513602 9942 raft_consensus.cc:482] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
W20240501 14:01:14.513725 4836 consensus_peers.cc:485] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.513839 9942 raft_consensus.cc:504] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:14.514559 9942 leader_election.cc:288] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:14.515486 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647f3f27faf435f81e076bfc2ef0795" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" is_pre_election: true
W20240501 14:01:14.516264 5099 leader_election.cc:334] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.516389 5099 leader_election.cc:302] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:14.517040 9609 raft_consensus.cc:2726] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:14.527572 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913ae5fda9e842e1b3a6d7dc418a5bfd" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:14.528407 5161 raft_consensus.cc:2454] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:14.529448 2080 leader_election.cc:302] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.529911 9696 raft_consensus.cc:2781] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.529981 9696 raft_consensus.cc:482] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.530043 9696 raft_consensus.cc:3037] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.532738 9696 raft_consensus.cc:504] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:14.533990 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913ae5fda9e842e1b3a6d7dc418a5bfd" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:14.534205 5163 raft_consensus.cc:3037] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.535990 4834 leader_election.cc:302] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.536469 9974 raft_consensus.cc:2781] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.536628 9974 raft_consensus.cc:686] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:14.536672 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea342bfcde3b4daa8dcc39126fd42cdc" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:14.536895 2406 raft_consensus.cc:2454] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:14.536862 9974 consensus_queue.cc:237] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
W20240501 14:01:14.537465 2342 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.5:40059, user_credentials={real_user=slave}} blocked reactor thread for 91695.3us
I20240501 14:01:14.538702 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:01:14.538882 2274 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.541570 2274 raft_consensus.cc:2454] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 2.
W20240501 14:01:14.542559 2082 leader_election.cc:334] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.543164 4834 leader_election.cc:302] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.542848 9696 leader_election.cc:288] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.543622 10004 raft_consensus.cc:2781] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.543715 10004 raft_consensus.cc:482] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.543817 10004 raft_consensus.cc:3037] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.548141 10004 raft_consensus.cc:504] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
W20240501 14:01:14.552947 4836 leader_election.cc:334] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.555277 10004 leader_election.cc:288] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:14.556114 2341 leader_election.cc:302] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters:
I20240501 14:01:14.556792 9872 raft_consensus.cc:2781] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.556933 9872 raft_consensus.cc:686] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:01:14.557057 9758 raft_consensus.cc:482] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.557118 9696 raft_consensus.cc:482] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.557250 9758 raft_consensus.cc:504] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.557220 9872 consensus_queue.cc:237] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:14.557255 9696 raft_consensus.cc:504] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:14.557888 9758 leader_election.cc:288] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
W20240501 14:01:14.559458 2082 leader_election.cc:334] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.559657 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:14.559866 5160 raft_consensus.cc:2454] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:14.559959 9696 leader_election.cc:288] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
W20240501 14:01:14.560170 4836 leader_election.cc:334] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.561003 4834 leader_election.cc:302] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.562978 10004 raft_consensus.cc:2781] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.563114 10004 raft_consensus.cc:482] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.563194 10004 raft_consensus.cc:3037] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.565766 2146 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:14.568192 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf051cecef424db5accfce66623346b5" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:14.568434 2406 raft_consensus.cc:2454] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:14.570359 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea342bfcde3b4daa8dcc39126fd42cdc" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:14.570647 2406 raft_consensus.cc:3037] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.573685 2406 raft_consensus.cc:2454] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:14.574229 9758 raft_consensus.cc:482] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.574429 9758 raft_consensus.cc:504] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:14.575138 9758 leader_election.cc:288] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:14.577327 4836 leader_election.cc:334] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.577387 10152 ts_tablet_manager.cc:902] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:14.579070 10004 raft_consensus.cc:504] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.580291 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:14.580526 5160 raft_consensus.cc:3037] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:14.581776 4836 leader_election.cc:334] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.583814 2004 catalog_manager.cc:5484] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.585682 5160 raft_consensus.cc:2454] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:14.587097 2080 leader_election.cc:302] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.589401 4834 leader_election.cc:302] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.590483 9931 raft_consensus.cc:2781] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.590606 9931 raft_consensus.cc:482] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.590675 9931 raft_consensus.cc:3037] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.593194 9931 raft_consensus.cc:504] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:14.594337 9931 leader_election.cc:288] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
W20240501 14:01:14.596143 2082 leader_election.cc:334] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.596915 4834 leader_election.cc:302] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.597571 9758 raft_consensus.cc:2781] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.597734 9758 raft_consensus.cc:686] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:14.598040 9758 consensus_queue.cc:237] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:14.598683 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf051cecef424db5accfce66623346b5" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:14.598894 2406 raft_consensus.cc:3037] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.603706 2406 raft_consensus.cc:2454] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:14.606595 9974 raft_consensus.cc:2781] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.618472 2080 leader_election.cc:302] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.619001 9696 raft_consensus.cc:2781] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.619134 9696 raft_consensus.cc:686] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:14.619496 9696 consensus_queue.cc:237] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:14.621798 5163 raft_consensus.cc:2454] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:14.631948 10004 leader_election.cc:288] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:14.657145 10165 ts_tablet_manager.cc:902] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:14.658020 10169 ts_tablet_manager.cc:902] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:14.659497 2080 leader_election.cc:302] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, d1a414d381f44829a2510cb76d481887; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.662660 9931 raft_consensus.cc:2781] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.662830 9931 raft_consensus.cc:686] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:14.663116 9931 consensus_queue.cc:237] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:14.667086 9974 raft_consensus.cc:686] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:14.667491 9974 consensus_queue.cc:237] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:14.668741 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647f3f27faf435f81e076bfc2ef0795" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:14.668959 5162 raft_consensus.cc:2454] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 1.
I20240501 14:01:14.669490 4834 leader_election.cc:302] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:14.670114 9875 raft_consensus.cc:2781] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:14.670241 9875 raft_consensus.cc:482] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f81f803ec7746ccb502c59b7cbf8eb2)
I20240501 14:01:14.670352 9875 raft_consensus.cc:3037] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.675940 9875 raft_consensus.cc:504] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:14.676795 9875 leader_election.cc:288] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:14.677248 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647f3f27faf435f81e076bfc2ef0795" candidate_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:14.677894 5163 raft_consensus.cc:3037] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:14.681035 5163 raft_consensus.cc:2454] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8c60f2bbac541dcba4424ab726f9ad9 in term 2.
I20240501 14:01:14.681604 4834 leader_election.cc:302] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9; no voters:
I20240501 14:01:14.682739 2274 raft_consensus.cc:1203] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:14.686925 10004 raft_consensus.cc:2781] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:14.687077 10004 raft_consensus.cc:686] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Becoming Leader. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Running, Role: LEADER
I20240501 14:01:14.687381 10004 consensus_queue.cc:237] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
W20240501 14:01:14.691957 4836 leader_election.cc:334] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.694228 10149 ts_tablet_manager.cc:902] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:14.701453 10140 tablet_copy_client.cc:308] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:01:14.702473 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 5c4312eec255417580288818bd559099 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:38303)
I20240501 14:01:14.702656 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 5c4312eec255417580288818bd559099 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:38303: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-5c4312eec255417580288818bd559099
I20240501 14:01:14.705639 2298 tablet_copy_source_session.cc:205] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.707340 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4312eec255417580288818bd559099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.711294 2004 catalog_manager.cc:5484] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:01:14.716835 2343 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.8:34221, user_credentials={real_user=slave}} blocked reactor thread for 270660us
I20240501 14:01:14.719986 2005 catalog_manager.cc:5484] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 2 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:14.728051 2000 catalog_manager.cc:5484] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:14.736287 5230 consensus_peers.cc:485] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.740374 2406 raft_consensus.cc:1261] T 0724625a6a8e4ce38e8087472b856c5c P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:14.746327 2533 consensus_queue.cc:237] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:14.750049 10186 ts_tablet_manager.cc:902] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:14.752493 10140 tablet_copy_client.cc:783] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.752756 10140 tablet_copy_client.cc:647] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.753427 10165 tablet_copy_client.cc:308] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
W20240501 14:01:14.753396 2343 leader_election.cc:334] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.754031 10169 tablet_copy_client.cc:308] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:01:14.754459 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet ccec6fa18f844ace900db8bfe91738a5 from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:51047)
I20240501 14:01:14.754612 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet ccec6fa18f844ace900db8bfe91738a5 from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:51047: session id = 745ac12076e84effa5bde1ecbd6a32ec-ccec6fa18f844ace900db8bfe91738a5
I20240501 14:01:14.754725 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet afc49574262e4819b6c86025c43c772a from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:49731)
I20240501 14:01:14.754902 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet afc49574262e4819b6c86025c43c772a from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:49731: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-afc49574262e4819b6c86025c43c772a
I20240501 14:01:14.757369 5184 tablet_copy_source_session.cc:205] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.757721 4921 tablet_copy_source_session.cc:205] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.759286 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc49574262e4819b6c86025c43c772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.759807 10115 tablet_copy_client.cc:523] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.763686 10115 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:14.763912 10165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccec6fa18f844ace900db8bfe91738a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.765728 2004 catalog_manager.cc:5484] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:14.766278 2004 catalog_manager.cc:5484] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.776495 9778 consensus_queue.cc:1035] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.777413 10140 tablet_copy_client.cc:523] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:01:14.780731 2343 leader_election.cc:334] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.782440 10152 tablet_copy_client.cc:308] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:14.783243 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet eb1fa60c64034b7d8c7b4879823b0d83 from peer d1a414d381f44829a2510cb76d481887 ({username='slave'} at 127.28.17.1:53971)
I20240501 14:01:14.783402 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet eb1fa60c64034b7d8c7b4879823b0d83 from peer d1a414d381f44829a2510cb76d481887 at {username='slave'} at 127.28.17.1:53971: session id = d1a414d381f44829a2510cb76d481887-eb1fa60c64034b7d8c7b4879823b0d83
I20240501 14:01:14.785982 2558 tablet_copy_source_session.cc:205] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.788349 2004 catalog_manager.cc:5484] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5). New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:14.783243 10149 tablet_copy_client.cc:308] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.7:41601
I20240501 14:01:14.792888 10152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1fa60c64034b7d8c7b4879823b0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.795989 10186 tablet_copy_client.cc:308] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:01:14.796670 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet efbd148f37bb46f0a4bf0fc562fff597 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:44257)
I20240501 14:01:14.796834 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet efbd148f37bb46f0a4bf0fc562fff597 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:44257: session id = a64f34785a7b44fa8070b0452a4e0ddd-efbd148f37bb46f0a4bf0fc562fff597
W20240501 14:01:14.797976 2471 consensus_peers.cc:485] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e081778a0f67486a904f3bed7dfe8644. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.799443 2298 tablet_copy_source_session.cc:205] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.800052 2000 catalog_manager.cc:5484] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 1 to 2, leader changed from 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 2 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:14.800820 10186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbd148f37bb46f0a4bf0fc562fff597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.801048 5184 tablet_copy_service.cc:140] P c9f94ddc5ff64ad79b9f01846a20e2fd: Received BeginTabletCopySession request for tablet 36f03024767442ac911a8a44f7ecdceb from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:51047)
I20240501 14:01:14.801209 5184 tablet_copy_service.cc:161] P c9f94ddc5ff64ad79b9f01846a20e2fd: Beginning new tablet copy session on tablet 36f03024767442ac911a8a44f7ecdceb from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:51047: session id = 745ac12076e84effa5bde1ecbd6a32ec-36f03024767442ac911a8a44f7ecdceb
I20240501 14:01:14.805706 5184 tablet_copy_source_session.cc:205] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:14.806555 10186 tablet_copy_client.cc:783] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.806788 10186 tablet_copy_client.cc:647] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.806819 10140 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:14.808144 10186 tablet_copy_client.cc:523] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.808480 10149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f03024767442ac911a8a44f7ecdceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:14.811568 10186 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:14.814193 10169 tablet_copy_client.cc:783] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.814564 10169 tablet_copy_client.cc:647] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.816817 10165 tablet_copy_client.cc:783] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.817039 10165 tablet_copy_client.cc:647] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.818867 10149 tablet_copy_client.cc:783] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.820084 10149 tablet_copy_client.cc:647] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:14.822152 5291 raft_consensus.cc:1261] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:14.822721 2473 consensus_peers.cc:485] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.823211 9465 consensus_queue.cc:1035] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.823578 10152 tablet_copy_client.cc:783] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:14.823774 10152 tablet_copy_client.cc:647] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: tablet copy: Starting download of 1 WAL segments...
W20240501 14:01:14.825995 2473 consensus_peers.cc:485] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.826123 10186 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.826452 10186 tablet_bootstrap.cc:492] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:01:14.826758 10186 ts_tablet_manager.cc:1367] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240501 14:01:14.827724 10186 raft_consensus.cc:348] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:14.827940 10186 raft_consensus.cc:729] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:01:14.828116 10186 consensus_queue.cc:260] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:14.828742 10186 ts_tablet_manager.cc:1398] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:14.829643 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-efbd148f37bb46f0a4bf0fc562fff597 received from {username='slave'} at 127.28.17.3:44257
I20240501 14:01:14.829754 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-efbd148f37bb46f0a4bf0fc562fff597 on tablet efbd148f37bb46f0a4bf0fc562fff597 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:14.832381 10152 tablet_copy_client.cc:523] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.835821 10152 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: Bootstrap starting.
I20240501 14:01:14.843712 5296 raft_consensus.cc:1203] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:14.850787 5296 raft_consensus.cc:1261] T b147c5f61a574c7d8a334b5f67ce732d P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:14.851518 9465 consensus_queue.cc:1035] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:14.853675 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.857241 10169 tablet_copy_client.cc:523] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.858119 10165 tablet_copy_client.cc:523] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.864850 10149 tablet_copy_client.cc:523] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:14.871168 10149 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:14.873839 2406 raft_consensus.cc:1261] T a7b11568c2d7420fa06f903e52723583 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:14.879762 9944 consensus_queue.cc:1035] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:14.885840 10165 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:14.890164 9465 raft_consensus.cc:2932] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:14.891417 5293 raft_consensus.cc:2932] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:14.899571 2000 catalog_manager.cc:5484] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.910996 10169 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:14.919212 10208 ts_tablet_manager.cc:902] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:14.928864 10152 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.929260 10152 tablet_bootstrap.cc:492] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: Bootstrap complete.
I20240501 14:01:14.929533 10152 ts_tablet_manager.cc:1367] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: Time spent bootstrapping tablet: real 0.094s user 0.049s sys 0.012s
I20240501 14:01:14.930253 10152 raft_consensus.cc:348] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:14.942065 5162 raft_consensus.cc:1261] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:14.944309 9875 consensus_queue.cc:1035] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:14.945276 10152 raft_consensus.cc:729] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1a414d381f44829a2510cb76d481887, State: Initialized, Role: LEARNER
W20240501 14:01:14.945483 4836 consensus_peers.cc:485] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.945559 10152 consensus_queue.cc:260] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: NON_VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: true } }
I20240501 14:01:14.948565 10152 ts_tablet_manager.cc:1398] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20240501 14:01:14.949340 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session d1a414d381f44829a2510cb76d481887-eb1fa60c64034b7d8c7b4879823b0d83 received from {username='slave'} at 127.28.17.1:53971
I20240501 14:01:14.949466 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session d1a414d381f44829a2510cb76d481887-eb1fa60c64034b7d8c7b4879823b0d83 on tablet eb1fa60c64034b7d8c7b4879823b0d83 with peer d1a414d381f44829a2510cb76d481887
I20240501 14:01:14.962832 10149 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.964533 10149 tablet_bootstrap.cc:492] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:14.969143 10149 ts_tablet_manager.cc:1367] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.098s user 0.010s sys 0.001s
I20240501 14:01:14.970712 10149 raft_consensus.cc:348] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
W20240501 14:01:14.973306 2213 consensus_peers.cc:485] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.974334 10149 raft_consensus.cc:729] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:14.974962 10149 consensus_queue.cc:260] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
W20240501 14:01:14.978252 2343 consensus_peers.cc:485] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:14.993736 10140 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:14.994163 10140 tablet_bootstrap.cc:492] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:01:14.995565 2408 raft_consensus.cc:1261] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:14.997619 9974 consensus_queue.cc:1035] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:15.006242 10140 ts_tablet_manager.cc:1367] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.199s user 0.016s sys 0.001s
I20240501 14:01:15.006490 10169 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:15.007082 10140 raft_consensus.cc:348] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:15.007337 10140 raft_consensus.cc:729] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:01:15.007524 10140 consensus_queue.cc:260] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:15.011513 10169 tablet_bootstrap.cc:492] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:01:15.019542 10169 ts_tablet_manager.cc:1367] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.001s
I20240501 14:01:15.022882 10149 ts_tablet_manager.cc:1398] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20240501 14:01:15.024526 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-36f03024767442ac911a8a44f7ecdceb received from {username='slave'} at 127.28.17.4:51047
I20240501 14:01:15.024653 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-36f03024767442ac911a8a44f7ecdceb on tablet 36f03024767442ac911a8a44f7ecdceb with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:15.031435 10169 raft_consensus.cc:348] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:15.033068 10169 raft_consensus.cc:729] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:01:15.033416 10169 consensus_queue.cc:260] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:15.034302 10169 ts_tablet_manager.cc:1398] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:01:15.036090 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-afc49574262e4819b6c86025c43c772a received from {username='slave'} at 127.28.17.7:49731
I20240501 14:01:15.036248 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-afc49574262e4819b6c86025c43c772a on tablet afc49574262e4819b6c86025c43c772a with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:15.041167 10208 tablet_copy_client.cc:308] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:01:15.042598 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet 6fe1b54933ad446082b1d2840b2c6e49 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:52743)
I20240501 14:01:15.042762 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet 6fe1b54933ad446082b1d2840b2c6e49 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:52743: session id = f8c60f2bbac541dcba4424ab726f9ad9-6fe1b54933ad446082b1d2840b2c6e49
I20240501 14:01:15.019593 10140 ts_tablet_manager.cc:1398] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 14:01:15.045012 2297 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-5c4312eec255417580288818bd559099 received from {username='slave'} at 127.28.17.7:38303
I20240501 14:01:15.045123 2297 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-5c4312eec255417580288818bd559099 on tablet 5c4312eec255417580288818bd559099 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:15.045409 2298 tablet_copy_source_session.cc:205] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:15.049999 2274 raft_consensus.cc:1261] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:15.050915 9814 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:15.053637 4836 consensus_peers.cc:485] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.053822 2146 raft_consensus.cc:1261] T d5086890bd254edebdf12812ff5c3728 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:15.054536 9814 consensus_queue.cc:1035] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:15.062848 2408 raft_consensus.cc:1203] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:15.080380 5162 raft_consensus.cc:1261] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:15.082412 9974 consensus_queue.cc:1035] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:15.091840 4836 consensus_peers.cc:485] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.094848 2406 raft_consensus.cc:1261] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240501 14:01:15.095052 2343 consensus_peers.cc:485] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.101387 10208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe1b54933ad446082b1d2840b2c6e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:15.097057 9974 consensus_queue.cc:1035] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240501 14:01:15.111452 2533 raft_consensus.cc:1203] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:01:15.113844 4836 consensus_peers.cc:485] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.119076 5164 raft_consensus.cc:1261] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:15.119874 9931 consensus_queue.cc:1035] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:15.120769 4899 consensus_queue.cc:237] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:15.125967 10165 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:15.126531 10165 tablet_bootstrap.cc:492] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:15.126854 10165 ts_tablet_manager.cc:1367] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.241s user 0.010s sys 0.000s
I20240501 14:01:15.127665 10165 raft_consensus.cc:348] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:15.127892 10165 raft_consensus.cc:729] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:15.128208 10165 consensus_queue.cc:260] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:15.129427 10165 ts_tablet_manager.cc:1398] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 14:01:15.133855 2082 consensus_peers.cc:485] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.141023 5184 tablet_copy_service.cc:342] P c9f94ddc5ff64ad79b9f01846a20e2fd: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-ccec6fa18f844ace900db8bfe91738a5 received from {username='slave'} at 127.28.17.4:51047
I20240501 14:01:15.141178 5184 tablet_copy_service.cc:433] P c9f94ddc5ff64ad79b9f01846a20e2fd: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-ccec6fa18f844ace900db8bfe91738a5 on tablet ccec6fa18f844ace900db8bfe91738a5 with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:15.142647 2145 raft_consensus.cc:1203] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:01:15.156461 4836 consensus_peers.cc:485] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:15.168365 2082 consensus_peers.cc:485] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.178613 10208 tablet_copy_client.cc:783] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:15.178936 10208 tablet_copy_client.cc:647] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:15.181645 10208 tablet_copy_client.cc:523] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:15.189419 5164 raft_consensus.cc:1203] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:15.192934 5164 raft_consensus.cc:1261] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:15.193604 9875 consensus_queue.cc:1035] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:15.203774 2406 raft_consensus.cc:1261] T bf051cecef424db5accfce66623346b5 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:01:15.204607 9931 consensus_queue.cc:1035] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:15.205021 10208 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:15.230373 9609 raft_consensus.cc:1050] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:15.231583 9609 consensus_queue.cc:237] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:15.237039 2277 raft_consensus.cc:1261] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.237083 5296 raft_consensus.cc:1261] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.237644 10245 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.238049 2533 raft_consensus.cc:1261] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.238658 9942 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.241806 9942 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.260466 9609 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.270332 10208 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:15.271040 2277 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.274641 10208 tablet_bootstrap.cc:492] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:01:15.274971 10208 ts_tablet_manager.cc:1367] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.001s
I20240501 14:01:15.276288 10208 raft_consensus.cc:348] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:15.275817 2533 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.276692 10208 raft_consensus.cc:729] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:01:15.276983 10208 consensus_queue.cc:260] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:15.277724 10208 ts_tablet_manager.cc:1398] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:15.279129 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-6fe1b54933ad446082b1d2840b2c6e49 received from {username='slave'} at 127.28.17.5:52743
I20240501 14:01:15.279238 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-6fe1b54933ad446082b1d2840b2c6e49 on tablet 6fe1b54933ad446082b1d2840b2c6e49 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:15.282855 2005 catalog_manager.cc:5484] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.289516 5164 consensus_queue.cc:237] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:15.290534 2146 raft_consensus.cc:1261] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:15.292048 2277 raft_consensus.cc:1261] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:15.292569 10167 consensus_queue.cc:1035] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:15.292641 9942 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:15.292881 4836 consensus_peers.cc:485] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.295331 2536 raft_consensus.cc:1261] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:15.295055 9609 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.295636 2277 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.299144 9942 consensus_queue.cc:1035] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:15.300554 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52eb73b91ed441199eade33e66b84b39 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:01:15.301671 4834 consensus_peers.cc:485] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9ccb0f16d8044da9f3f353ffbea5ce9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.302537 2536 raft_consensus.cc:2932] T 52eb73b91ed441199eade33e66b84b39 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.303257 2001 catalog_manager.cc:5484] T 52eb73b91ed441199eade33e66b84b39 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:15.311496 10167 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:15.315168 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 52eb73b91ed441199eade33e66b84b39 with delete_type TABLET_DATA_TOMBSTONED (TS 5a4b9a96e9734baa8a03d6056bfa934c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36154
I20240501 14:01:15.329643 2146 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } } }
I20240501 14:01:15.332967 2001 catalog_manager.cc:5484] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:15.346766 4899 consensus_queue.cc:237] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:15.348014 10115 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:15.348459 10115 tablet_bootstrap.cc:492] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:15.349063 10251 ts_tablet_manager.cc:902] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:15.349858 10252 tablet_replica.cc:330] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: stopping tablet replica
I20240501 14:01:15.350069 10252 raft_consensus.cc:2229] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:15.350217 10252 pending_rounds.cc:62] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Trying to abort 1 pending ops.
I20240501 14:01:15.350260 10252 pending_rounds.cc:68] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022887220145225728 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "52eb73b91ed441199eade33e66b84b39" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } } }
I20240501 14:01:15.350646 10252 raft_consensus.cc:2894] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:01:15.350764 10252 raft_consensus.cc:2866] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:01:15.350872 10252 raft_consensus.cc:2258] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:15.351095 10115 ts_tablet_manager.cc:1367] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.587s user 0.050s sys 0.005s
I20240501 14:01:15.351872 10115 raft_consensus.cc:348] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:15.352085 5162 raft_consensus.cc:1261] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:15.352136 10115 raft_consensus.cc:729] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
W20240501 14:01:15.352263 4836 consensus_peers.cc:485] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.352332 10115 consensus_queue.cc:260] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:15.352885 10115 ts_tablet_manager.cc:1398] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:15.353375 5164 raft_consensus.cc:1203] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:15.353325 10004 consensus_queue.cc:1035] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.355008 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-b9d8d993ce834c1db4dd4418f7a85780 received from {username='slave'} at 127.28.17.8:53689
I20240501 14:01:15.355129 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-b9d8d993ce834c1db4dd4418f7a85780 on tablet b9d8d993ce834c1db4dd4418f7a85780 with peer 5a4b9a96e9734baa8a03d6056bfa934c
W20240501 14:01:15.359723 4834 consensus_peers.cc:485] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e605c78442442dd82c598bac2fd84cd. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:15.367081 2343 consensus_peers.cc:485] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.367095 5296 raft_consensus.cc:1203] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:01:15.368203 10249 raft_consensus.cc:1050] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:15.368988 10249 consensus_queue.cc:237] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:15.377966 9974 raft_consensus.cc:2932] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:15.381295 10252 ts_tablet_manager.cc:1875] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:15.387332 5296 raft_consensus.cc:1261] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.388310 9942 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.384085 5164 raft_consensus.cc:2932] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:15.391570 10252 ts_tablet_manager.cc:1888] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:15.392154 10252 log.cc:1196] T 52eb73b91ed441199eade33e66b84b39 P 5a4b9a96e9734baa8a03d6056bfa934c: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/wals/52eb73b91ed441199eade33e66b84b39
I20240501 14:01:15.393508 1992 catalog_manager.cc:4830] TS 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): tablet 52eb73b91ed441199eade33e66b84b39 (table rebalancer_extra_table_24 [id=189c751a391d46929caa5ca489d9d9bb]) successfully deleted
I20240501 14:01:15.395342 2005 catalog_manager.cc:5484] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 3, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:15.397069 4899 raft_consensus.cc:1261] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.397665 9942 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.409574 2146 raft_consensus.cc:1261] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.410202 10249 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.413314 9942 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.413867 2146 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.419306 5296 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.420274 2001 catalog_manager.cc:5484] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:15.425341 4901 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
W20240501 14:01:15.433972 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:15.448118 5164 consensus_queue.cc:237] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:15.449880 10251 tablet_copy_client.cc:308] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:15.450114 4901 raft_consensus.cc:1261] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:15.450577 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet e081778a0f67486a904f3bed7dfe8644 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35609)
I20240501 14:01:15.450709 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet e081778a0f67486a904f3bed7dfe8644 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35609: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-e081778a0f67486a904f3bed7dfe8644
I20240501 14:01:15.453399 2558 tablet_copy_source_session.cc:205] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:15.456640 9609 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:15.465569 10245 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.469512 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ff679e55a5446099efbdbcd1109f245 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:15.470368 4901 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.472705 2000 catalog_manager.cc:5484] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:15.475833 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e081778a0f67486a904f3bed7dfe8644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:15.477238 2536 raft_consensus.cc:1203] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:15.485563 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ff679e55a5446099efbdbcd1109f245 with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48706
I20240501 14:01:15.504601 10251 tablet_copy_client.cc:783] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:15.505249 10251 tablet_copy_client.cc:647] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:15.507571 5291 consensus_queue.cc:237] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:15.509532 5296 raft_consensus.cc:1261] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:15.510268 5228 consensus_peers.cc:485] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c88cc21c0e0743999ee493f5ee856aef. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.510625 10251 tablet_copy_client.cc:523] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:15.510679 2406 raft_consensus.cc:1261] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:15.510833 4901 raft_consensus.cc:1261] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:15.511380 9778 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:15.511200 10245 consensus_queue.cc:1035] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:15.511670 10190 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:15.512495 10272 tablet_replica.cc:330] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:01:15.512709 10272 raft_consensus.cc:2229] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:15.512955 10272 raft_consensus.cc:2258] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:15.513549 5291 raft_consensus.cc:2932] T 0ff679e55a5446099efbdbcd1109f245 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.515997 10272 ts_tablet_manager.cc:1875] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:15.527494 10251 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:15.530364 9778 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:15.531483 2406 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:15.535209 4901 raft_consensus.cc:1203] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:15.539685 2004 catalog_manager.cc:5484] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:15.547894 4901 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
W20240501 14:01:15.558887 2473 consensus_peers.cc:485] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.567579 9974 raft_consensus.cc:1050] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:01:15.568542 9974 consensus_queue.cc:237] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:15.572561 10251 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:15.573009 10251 tablet_bootstrap.cc:492] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:15.573272 10251 ts_tablet_manager.cc:1367] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.018s
I20240501 14:01:15.573921 10251 raft_consensus.cc:348] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:15.574154 10251 raft_consensus.cc:729] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:15.574337 10251 consensus_queue.cc:260] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:15.576594 10251 ts_tablet_manager.cc:1398] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:15.577330 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-e081778a0f67486a904f3bed7dfe8644 received from {username='slave'} at 127.28.17.2:35609
I20240501 14:01:15.577441 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-e081778a0f67486a904f3bed7dfe8644 on tablet e081778a0f67486a904f3bed7dfe8644 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:15.582281 10272 ts_tablet_manager.cc:1888] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:15.582422 10272 log.cc:1196] T 0ff679e55a5446099efbdbcd1109f245 P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/0ff679e55a5446099efbdbcd1109f245
I20240501 14:01:15.585417 5164 raft_consensus.cc:1261] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.586148 9758 consensus_queue.cc:1035] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.586684 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet 0ff679e55a5446099efbdbcd1109f245 (table rebalancer_extra_table_52 [id=7a56a59140c34161b5ee372201ccd146]) successfully deleted
I20240501 14:01:15.591647 2146 raft_consensus.cc:1261] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240501 14:01:15.592569 4836 consensus_peers.cc:485] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.592875 9758 consensus_queue.cc:1035] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.609131 9758 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:15.616397 5163 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:15.619786 2000 catalog_manager.cc:5484] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:15.621840 2145 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:15.627066 10231 raft_consensus.cc:1050] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER d1a414d381f44829a2510cb76d481887 to VOTER
I20240501 14:01:15.627877 10231 consensus_queue.cc:237] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:15.637022 4901 consensus_queue.cc:237] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:15.641121 10245 raft_consensus.cc:1050] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:15.642017 10245 consensus_queue.cc:237] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:15.644840 2146 raft_consensus.cc:1261] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:15.644925 5163 raft_consensus.cc:1261] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:15.646135 4836 consensus_peers.cc:485] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.647125 9758 consensus_queue.cc:1035] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:15.647768 9758 consensus_queue.cc:1035] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:15.651168 10004 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:15.655895 2536 raft_consensus.cc:1261] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.656616 9942 consensus_queue.cc:1035] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.657248 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afc49574262e4819b6c86025c43c772a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:15.659948 9264 raft_consensus.cc:1050] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:01:15.659884 2146 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
W20240501 14:01:15.660845 5099 consensus_peers.cc:485] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.660760 9264 consensus_queue.cc:237] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:15.665115 2000 catalog_manager.cc:5484] T afc49574262e4819b6c86025c43c772a P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:15.668717 5163 raft_consensus.cc:2932] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:15.673610 5296 raft_consensus.cc:1261] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.675051 10249 consensus_queue.cc:1035] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.691370 10291 ts_tablet_manager.cc:902] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:15.695889 4901 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 14:01:15.696297 2473 consensus_peers.cc:485] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.697114 10228 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:15.697805 2146 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:15.699903 5163 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:01:15.701469 10228 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:01:15.701752 10234 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:01:15.731683 2343 consensus_peers.cc:485] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:15.739612 2082 consensus_peers.cc:485] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.748670 10245 raft_consensus.cc:2932] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
W20240501 14:01:15.755672 2343 consensus_peers.cc:485] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.757623 2536 raft_consensus.cc:2932] T 36f03024767442ac911a8a44f7ecdceb P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:15.757900 5293 raft_consensus.cc:2932] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:15.765539 2004 catalog_manager.cc:5484] T 36f03024767442ac911a8a44f7ecdceb P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
W20240501 14:01:15.775590 5230 consensus_peers.cc:485] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:15.777140 2082 consensus_peers.cc:485] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:15.778438 1992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:15.781723 2406 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.782878 10234 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.783869 9945 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.791219 2146 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.794185 2536 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.795127 9945 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.802013 5164 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.802702 9944 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.808787 2000 catalog_manager.cc:5484] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.808941 4899 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.810154 10138 raft_consensus.cc:1050] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:15.814491 10138 consensus_queue.cc:237] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:15.815184 9944 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:15.820910 2407 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:15.822170 2536 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:15.831261 2000 catalog_manager.cc:5484] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:01:15.837457 2213 consensus_peers.cc:485] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:15.839354 1992 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.17.6:46037, user_credentials={real_user=slave}} blocked reactor thread for 60958.9us
W20240501 14:01:15.840574 1992 catalog_manager.cc:4573] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet afc49574262e4819b6c86025c43c772a: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:15.841035 10307 raft_consensus.cc:1050] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:01:15.841809 10307 consensus_queue.cc:237] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:15.843461 5164 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d1a414d381f44829a2510cb76d481887 (127.28.17.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.844784 5163 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
W20240501 14:01:15.854034 2343 consensus_peers.cc:485] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.859131 10311 ts_tablet_manager.cc:902] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:15.859033 2533 consensus_queue.cc:237] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:15.859405 5293 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:15.859927 10147 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:15.861398 2146 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:15.862037 10228 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:15.864858 2274 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:15.865520 10234 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.867345 5163 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:01:15.867118 2145 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.870906 10147 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:01:15.871587 10147 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:01:15.873358 2213 consensus_peers.cc:485] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.874250 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb1fa60c64034b7d8c7b4879823b0d83 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:15.878594 2005 catalog_manager.cc:5484] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:15.882865 9942 raft_consensus.cc:1050] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:15.883736 9942 consensus_queue.cc:237] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:15.884867 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1fa60c64034b7d8c7b4879823b0d83 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:15.889600 2536 raft_consensus.cc:1261] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.890413 10245 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.891755 2406 raft_consensus.cc:1261] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.893077 9609 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.888765 10316 tablet_replica.cc:330] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:15.893595 10316 raft_consensus.cc:2229] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:15.893796 10316 raft_consensus.cc:2258] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:15.895555 10316 ts_tablet_manager.cc:1875] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:15.901897 2274 raft_consensus.cc:1203] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:15.908227 9872 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.909531 10316 ts_tablet_manager.cc:1888] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:01:15.909636 10316 log.cc:1196] T eb1fa60c64034b7d8c7b4879823b0d83 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/eb1fa60c64034b7d8c7b4879823b0d83
I20240501 14:01:15.911088 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet eb1fa60c64034b7d8c7b4879823b0d83 (table rebalancer_extra_table_4 [id=85455387e201473b9e5fdcf58be54e10]) successfully deleted
I20240501 14:01:15.912097 5291 raft_consensus.cc:1261] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.913166 5162 raft_consensus.cc:1261] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:15.914649 5162 raft_consensus.cc:1261] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.915284 5162 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.915709 5294 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.917605 2274 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.919322 10308 consensus_queue.cc:1035] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.920454 9264 consensus_queue.cc:1035] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:01:15.921411 2213 consensus_peers.cc:485] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:15.922101 5291 raft_consensus.cc:1261] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:15.926414 2000 catalog_manager.cc:5484] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 7 to 8, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:15.931910 2407 consensus_queue.cc:237] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
W20240501 14:01:15.936944 2213 consensus_peers.cc:485] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:15.937548 2213 consensus_peers.cc:485] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.937814 5291 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:15.938453 9814 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:15.943081 5162 raft_consensus.cc:1261] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:01:15.943734 9814 consensus_queue.cc:1035] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:01:15.946323 10245 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:15.946610 10228 consensus_queue.cc:1035] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:15.954713 10291 tablet_copy_client.cc:308] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:01:15.954851 5164 raft_consensus.cc:2932] T eb1fa60c64034b7d8c7b4879823b0d83 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } } }
I20240501 14:01:15.958453 10249 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:15.961354 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet b9ccb0f16d8044da9f3f353ffbea5ce9 from peer a64f34785a7b44fa8070b0452a4e0ddd ({username='slave'} at 127.28.17.3:36677)
I20240501 14:01:15.961519 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet b9ccb0f16d8044da9f3f353ffbea5ce9 from peer a64f34785a7b44fa8070b0452a4e0ddd at {username='slave'} at 127.28.17.3:36677: session id = a64f34785a7b44fa8070b0452a4e0ddd-b9ccb0f16d8044da9f3f353ffbea5ce9
I20240501 14:01:15.964231 4921 tablet_copy_source_session.cc:205] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:15.963940 2406 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:15.966521 10291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ccb0f16d8044da9f3f353ffbea5ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:15.969684 9872 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.970430 5162 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.973448 5291 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
W20240501 14:01:15.974750 5099 consensus_peers.cc:485] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.975167 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9d8d993ce834c1db4dd4418f7a85780 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:01:15.977802 2343 consensus_peers.cc:485] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.978976 9945 raft_consensus.cc:1050] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:01:15.979766 9945 consensus_queue.cc:237] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:15.979825 5293 raft_consensus.cc:2932] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:15.983675 2000 catalog_manager.cc:5484] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:15.987140 9264 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:15.987576 2533 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:15.990235 5291 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
W20240501 14:01:15.994755 2213 consensus_peers.cc:485] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:15.996518 5162 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:15.998903 2000 catalog_manager.cc:5484] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:16.000268 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:16.008095 2213 consensus_peers.cc:485] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.009829 4901 raft_consensus.cc:1261] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:16.011642 2145 raft_consensus.cc:1261] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:16.012214 9264 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:16.014644 2001 catalog_manager.cc:5484] T b9d8d993ce834c1db4dd4418f7a85780 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 8 to 9, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:16.016264 5162 raft_consensus.cc:1261] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:16.017235 10309 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:16.018460 9264 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:16.019464 10291 tablet_copy_client.cc:783] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:16.019866 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d8d993ce834c1db4dd4418f7a85780 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35676
W20240501 14:01:16.020509 2082 consensus_peers.cc:485] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.020726 10291 tablet_copy_client.cc:647] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:16.021442 2274 consensus_queue.cc:237] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:16.023402 10291 tablet_copy_client.cc:523] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:16.023933 5163 raft_consensus.cc:1261] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.024597 9264 consensus_queue.cc:1035] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:16.026372 5230 consensus_peers.cc:485] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.027011 9264 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:16.027540 5163 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:16.031154 10311 tablet_copy_client.cc:308] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:01:16.031849 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet 8e605c78442442dd82c598bac2fd84cd from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:39663)
I20240501 14:01:16.031982 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet 8e605c78442442dd82c598bac2fd84cd from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:39663: session id = 745ac12076e84effa5bde1ecbd6a32ec-8e605c78442442dd82c598bac2fd84cd
I20240501 14:01:16.037091 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c4312eec255417580288818bd559099 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:16.039002 4921 tablet_copy_source_session.cc:205] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:16.039284 2004 catalog_manager.cc:5484] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:01:16.044492 2343 consensus_peers.cc:485] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:16.045714 4836 consensus_peers.cc:485] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.046396 5293 raft_consensus.cc:1261] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.046942 10311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e605c78442442dd82c598bac2fd84cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:16.047060 10307 consensus_queue.cc:1035] T 5c4312eec255417580288818bd559099 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.049275 5293 raft_consensus.cc:2932] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:16.062793 10331 tablet_replica.cc:330] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:01:16.063128 10331 raft_consensus.cc:2229] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:16.063702 10291 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap starting.
I20240501 14:01:16.064214 4899 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
W20240501 14:01:16.064873 2473 consensus_peers.cc:485] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.064621 5163 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:16.066360 5162 consensus_queue.cc:237] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:16.063166 9945 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:16.069212 10331 raft_consensus.cc:2258] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:01:16.073527 4836 consensus_peers.cc:485] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:16.074105 2082 consensus_peers.cc:485] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.075724 2536 raft_consensus.cc:1261] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.076320 9942 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.077114 10311 tablet_copy_client.cc:783] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:16.077471 10311 tablet_copy_client.cc:647] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:16.079288 9942 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:16.084395 2005 catalog_manager.cc:5484] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:16.085659 10311 tablet_copy_client.cc:523] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:16.091346 2536 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:16.094712 2143 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:16.096776 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccec6fa18f844ace900db8bfe91738a5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:01:16.100759 5099 consensus_peers.cc:485] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.102180 5291 raft_consensus.cc:1261] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.103132 10327 consensus_queue.cc:1035] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.105235 2005 catalog_manager.cc:5484] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New cstate: current_term: 1 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:16.108136 10331 ts_tablet_manager.cc:1875] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:16.109814 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet ccec6fa18f844ace900db8bfe91738a5 with delete_type TABLET_DATA_TOMBSTONED (TS a64f34785a7b44fa8070b0452a4e0ddd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51622
I20240501 14:01:16.112021 10311 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:16.116091 5291 raft_consensus.cc:2932] T ccec6fa18f844ace900db8bfe91738a5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:16.119330 10291 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:16.119843 10291 tablet_bootstrap.cc:492] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: Bootstrap complete.
I20240501 14:01:16.121045 10291 ts_tablet_manager.cc:1367] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.000s
I20240501 14:01:16.121915 10291 raft_consensus.cc:348] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:16.123160 10291 raft_consensus.cc:729] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Initialized, Role: LEARNER
I20240501 14:01:16.124105 10291 consensus_queue.cc:260] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: NON_VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: true } }
I20240501 14:01:16.128831 10339 ts_tablet_manager.cc:902] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:16.130601 2277 consensus_queue.cc:237] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:16.133235 4899 raft_consensus.cc:1261] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.134035 10307 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.137037 2143 raft_consensus.cc:1261] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.138288 10338 tablet_replica.cc:330] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: stopping tablet replica
I20240501 14:01:16.138480 10291 ts_tablet_manager.cc:1398] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240501 14:01:16.139463 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-b9ccb0f16d8044da9f3f353ffbea5ce9 received from {username='slave'} at 127.28.17.3:36677
I20240501 14:01:16.139590 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session a64f34785a7b44fa8070b0452a4e0ddd-b9ccb0f16d8044da9f3f353ffbea5ce9 on tablet b9ccb0f16d8044da9f3f353ffbea5ce9 with peer a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:16.141340 10307 consensus_queue.cc:1035] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.143474 10338 raft_consensus.cc:2229] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:16.143765 10338 raft_consensus.cc:2258] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:01:16.144836 2473 consensus_peers.cc:485] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.144882 10338 ts_tablet_manager.cc:1875] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:16.150622 9264 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:16.152812 10331 ts_tablet_manager.cc:1888] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:01:16.152925 10331 log.cc:1196] T b9d8d993ce834c1db4dd4418f7a85780 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/b9d8d993ce834c1db4dd4418f7a85780
I20240501 14:01:16.155818 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6fe1b54933ad446082b1d2840b2c6e49 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:16.156136 4899 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:16.162351 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet b9d8d993ce834c1db4dd4418f7a85780 (table rebalance_test_table_7 [id=0e200730a8f241d09fb23172cd161e14]) successfully deleted
I20240501 14:01:16.162914 10311 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:16.163410 10311 tablet_bootstrap.cc:492] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:16.163718 10311 ts_tablet_manager.cc:1367] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.000s
I20240501 14:01:16.164947 10311 raft_consensus.cc:348] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:16.165376 10311 raft_consensus.cc:729] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:16.165644 10311 consensus_queue.cc:260] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:16.166970 2143 raft_consensus.cc:2932] T 6fe1b54933ad446082b1d2840b2c6e49 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:16.167465 2005 catalog_manager.cc:5484] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:16.174854 10311 ts_tablet_manager.cc:1398] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:01:16.175431 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe1b54933ad446082b1d2840b2c6e49 with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:16.182271 10338 ts_tablet_manager.cc:1888] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:16.183131 10338 log.cc:1196] T ccec6fa18f844ace900db8bfe91738a5 P a64f34785a7b44fa8070b0452a4e0ddd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/wals/ccec6fa18f844ace900db8bfe91738a5
I20240501 14:01:16.183763 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-8e605c78442442dd82c598bac2fd84cd received from {username='slave'} at 127.28.17.4:39663
I20240501 14:01:16.183884 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-8e605c78442442dd82c598bac2fd84cd on tablet 8e605c78442442dd82c598bac2fd84cd with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:16.184381 1990 catalog_manager.cc:4830] TS a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): tablet ccec6fa18f844ace900db8bfe91738a5 (table rebalancer_extra_table_40 [id=a865347ff7654d7e9cbcf7b4492e7ea2]) successfully deleted
I20240501 14:01:16.190759 10339 tablet_copy_client.cc:308] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:01:16.192431 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet c88cc21c0e0743999ee493f5ee856aef from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35917)
I20240501 14:01:16.192636 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet c88cc21c0e0743999ee493f5ee856aef from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35917: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-c88cc21c0e0743999ee493f5ee856aef
I20240501 14:01:16.195822 5316 tablet_copy_source_session.cc:205] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:01:16.199379 4836 consensus_peers.cc:485] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.201285 10349 tablet_replica.cc:330] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:16.201519 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88cc21c0e0743999ee493f5ee856aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:16.202190 10349 raft_consensus.cc:2229] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:16.202873 10349 raft_consensus.cc:2258] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:16.204744 10349 ts_tablet_manager.cc:1875] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:16.212631 10349 ts_tablet_manager.cc:1888] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:16.212760 10349 log.cc:1196] T 6fe1b54933ad446082b1d2840b2c6e49 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/6fe1b54933ad446082b1d2840b2c6e49
I20240501 14:01:16.212795 10339 tablet_copy_client.cc:783] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:16.212991 10339 tablet_copy_client.cc:647] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:16.214334 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 6fe1b54933ad446082b1d2840b2c6e49 (table rebalancer_extra_table_32 [id=a9a9029286c5419599701a9993b3f188]) successfully deleted
I20240501 14:01:16.215260 10339 tablet_copy_client.cc:523] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:16.218492 10339 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:16.224045 10339 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:16.224368 10339 tablet_bootstrap.cc:492] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:16.224611 10339 ts_tablet_manager.cc:1367] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:01:16.225145 10339 raft_consensus.cc:348] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:16.225368 10339 raft_consensus.cc:729] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:16.225580 10339 consensus_queue.cc:260] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:16.226287 10339 ts_tablet_manager.cc:1398] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 14:01:16.227324 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-c88cc21c0e0743999ee493f5ee856aef received from {username='slave'} at 127.28.17.2:35917
I20240501 14:01:16.227447 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-c88cc21c0e0743999ee493f5ee856aef on tablet c88cc21c0e0743999ee493f5ee856aef with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:16.248406 2405 raft_consensus.cc:1203] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:16.269255 2277 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597"
dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35158
I20240501 14:01:16.269515 2277 raft_consensus.cc:595] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Received request to transfer leadership
I20240501 14:01:16.272660 10308 raft_consensus.cc:979] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: : Instructing follower c9f94ddc5ff64ad79b9f01846a20e2fd to start an election
I20240501 14:01:16.272805 10308 raft_consensus.cc:1067] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Signalling peer c9f94ddc5ff64ad79b9f01846a20e2fd to start an election
I20240501 14:01:16.273633 5164 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597"
dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
from {username='slave'} at 127.28.17.2:40205
I20240501 14:01:16.273844 5164 raft_consensus.cc:482] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:16.273938 5164 raft_consensus.cc:3037] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:16.276618 5164 raft_consensus.cc:504] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:16.277501 5164 leader_election.cc:288] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:16.278383 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
I20240501 14:01:16.278411 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:16.278602 2277 raft_consensus.cc:3032] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:16.278648 2536 raft_consensus.cc:3037] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:16.278669 2277 raft_consensus.cc:729] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:01:16.278595 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbd148f37bb46f0a4bf0fc562fff597" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:16.278766 2405 raft_consensus.cc:3037] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:16.278947 2277 consensus_queue.cc:260] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:16.279304 2277 raft_consensus.cc:3037] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:16.281235 2277 raft_consensus.cc:2454] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 2.
I20240501 14:01:16.281314 2536 raft_consensus.cc:2454] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 2.
I20240501 14:01:16.281296 2405 raft_consensus.cc:2454] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 2.
I20240501 14:01:16.281980 5097 leader_election.cc:302] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
I20240501 14:01:16.282280 9942 raft_consensus.cc:2781] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:16.282389 9942 raft_consensus.cc:686] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:01:16.282666 9942 consensus_queue.cc:237] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:16.287655 2000 catalog_manager.cc:5484] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 1 to 2, leader changed from b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 2 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:16.343376 2082 consensus_peers.cc:485] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:16.351426 5230 consensus_peers.cc:485] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:16.366212 2082 consensus_peers.cc:485] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:16.386287 10231 raft_consensus.cc:1050] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:16.387056 10231 consensus_queue.cc:237] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:16.392829 2277 raft_consensus.cc:1261] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:16.393744 5291 raft_consensus.cc:1261] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:16.394603 10234 consensus_queue.cc:1035] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:16.395043 10234 consensus_queue.cc:1035] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:01:16.395471 2473 consensus_peers.cc:485] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.408866 10228 raft_consensus.cc:2932] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:16.413811 2277 raft_consensus.cc:2932] T e081778a0f67486a904f3bed7dfe8644 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:16.416095 5291 raft_consensus.cc:2932] T e081778a0f67486a904f3bed7dfe8644 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:16.418397 2000 catalog_manager.cc:5484] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:16.434684 2536 raft_consensus.cc:1203] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
W20240501 14:01:16.559010 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:16.569566 2277 consensus_queue.cc:237] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:16.573571 5295 raft_consensus.cc:1261] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:16.573879 2211 consensus_peers.cc:485] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f30705f9c298491dbfb32bdc554647e4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.575233 9945 consensus_queue.cc:1035] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:16.575790 2213 consensus_peers.cc:485] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.578789 10308 raft_consensus.cc:2932] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:16.579632 5295 raft_consensus.cc:2932] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } } }
I20240501 14:01:16.586341 2005 catalog_manager.cc:5484] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index -1 to 2, NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) added. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:16.607211 2536 consensus_queue.cc:237] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:16.609380 2277 raft_consensus.cc:1261] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.609416 2145 raft_consensus.cc:1261] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:16.609665 2473 consensus_peers.cc:485] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4a0a92e0d48493fac254f1b249a63cf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.609941 10231 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.610229 10234 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.613359 10231 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:16.613821 2145 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:16.613844 2274 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:16.621429 2005 catalog_manager.cc:5484] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 2, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:16.634063 2274 raft_consensus.cc:1203] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:16.653061 4899 consensus_queue.cc:237] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
W20240501 14:01:16.659965 4834 consensus_peers.cc:485] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a74335efeb04521baf1720a14093643. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.661063 2145 raft_consensus.cc:1261] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:16.661199 4836 consensus_peers.cc:485] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.661744 10419 consensus_queue.cc:1035] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.676280 9974 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:16.680572 2145 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:16.689018 2000 catalog_manager.cc:5484] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 reported cstate change: config changed from index -1 to 2, NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) added. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } } }
I20240501 14:01:16.735337 2536 consensus_queue.cc:237] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:16.737900 5162 raft_consensus.cc:1261] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.737905 2406 raft_consensus.cc:1261] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:16.738157 2472 consensus_peers.cc:485] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 205af5386f904e70989e647da22750e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.738463 10447 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.738780 10231 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.741760 10228 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:16.745206 2406 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:16.749573 2004 catalog_manager.cc:5484] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:16.766628 2536 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:16.768105 10327 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.768227 5162 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:16.775835 10419 raft_consensus.cc:1050] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER a64f34785a7b44fa8070b0452a4e0ddd to VOTER
I20240501 14:01:16.776679 10419 consensus_queue.cc:237] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:16.780444 2405 consensus_queue.cc:237] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:16.782205 2145 raft_consensus.cc:1261] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:16.783929 10004 consensus_queue.cc:1035] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:16.784675 2274 raft_consensus.cc:1261] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:16.785212 10138 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:01:16.785883 2343 consensus_peers.cc:485] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221): Couldn't send request to peer 5a4b9a96e9734baa8a03d6056bfa934c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67da9784350e49209e979e232a79cfc1. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:16.786540 2343 consensus_peers.cc:485] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.786971 2406 raft_consensus.cc:1261] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:16.787415 10004 consensus_queue.cc:1035] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:16.787541 10470 raft_consensus.cc:2932] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:16.788437 2274 raft_consensus.cc:2932] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } } }
I20240501 14:01:16.790191 10004 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
W20240501 14:01:16.790688 4836 consensus_peers.cc:485] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.791492 2406 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.795156 2000 catalog_manager.cc:5484] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index -1 to 3, NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) added. New cstate: current_term: 2 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:16.794996 2145 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.801872 2005 catalog_manager.cc:5484] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:16.808727 4899 consensus_queue.cc:237] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:16.811218 2406 raft_consensus.cc:1261] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.811264 2145 raft_consensus.cc:1261] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:16.813622 10420 consensus_queue.cc:1035] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.814097 10420 consensus_queue.cc:1035] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.822074 10004 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.825927 2143 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.826488 2406 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:16.827051 9942 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.830850 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9ccb0f16d8044da9f3f353ffbea5ce9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:16.830771 2406 raft_consensus.cc:2932] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.844400 2004 catalog_manager.cc:5484] T b9ccb0f16d8044da9f3f353ffbea5ce9 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:01:16.859771 1992 catalog_manager.cc:4573] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b9ccb0f16d8044da9f3f353ffbea5ce9: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:16.868813 5295 consensus_queue.cc:237] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:16.874044 2274 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:16.874663 4899 raft_consensus.cc:1261] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.875298 10190 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:16.875764 10327 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:16.877424 2405 raft_consensus.cc:1261] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:16.881145 10190 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:16.881773 5228 consensus_peers.cc:485] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6904a974640a4e84a80b1fb0749365b7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:16.885377 10378 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:16.886008 4901 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:16.892920 2005 catalog_manager.cc:5484] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:16.896680 5162 consensus_queue.cc:237] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:16.900115 2407 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:16.904531 10327 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:16.911401 10327 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.912619 2407 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.916705 2536 raft_consensus.cc:1261] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:01:16.923547 9942 consensus_queue.cc:1035] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:16.925592 2004 catalog_manager.cc:5484] T efbd148f37bb46f0a4bf0fc562fff597 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 3 to 5, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New cstate: current_term: 2 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.930707 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet efbd148f37bb46f0a4bf0fc562fff597 with delete_type TABLET_DATA_TOMBSTONED (TS b3da1f36f37e4a1aa9f280c2fd0e1fba not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35676
I20240501 14:01:16.949703 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efbd148f37bb46f0a4bf0fc562fff597 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:16.952286 2536 raft_consensus.cc:2932] T efbd148f37bb46f0a4bf0fc562fff597 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } } }
I20240501 14:01:16.954594 2405 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:16.970185 10494 tablet_replica.cc:330] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: stopping tablet replica
I20240501 14:01:16.970476 10494 raft_consensus.cc:2229] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:16.970793 10494 raft_consensus.cc:2258] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 14:01:16.972916 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:16.974689 10494 ts_tablet_manager.cc:1875] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:16.981731 10494 ts_tablet_manager.cc:1888] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 14:01:16.982020 10494 log.cc:1196] T efbd148f37bb46f0a4bf0fc562fff597 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/wals/efbd148f37bb46f0a4bf0fc562fff597
I20240501 14:01:16.983188 1990 catalog_manager.cc:4830] TS b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): tablet efbd148f37bb46f0a4bf0fc562fff597 (table rebalancer_extra_table_18 [id=8e8d2cf373104443ab0cf1d81db160b0]) successfully deleted
I20240501 14:01:16.986197 10500 ts_tablet_manager.cc:902] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Initiating tablet copy from peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:17.022598 10500 tablet_copy_client.cc:308] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.2:46301
I20240501 14:01:17.023217 2298 tablet_copy_service.cc:140] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Received BeginTabletCopySession request for tablet f30705f9c298491dbfb32bdc554647e4 from peer c9f94ddc5ff64ad79b9f01846a20e2fd ({username='slave'} at 127.28.17.7:38303)
I20240501 14:01:17.023362 2298 tablet_copy_service.cc:161] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Beginning new tablet copy session on tablet f30705f9c298491dbfb32bdc554647e4 from peer c9f94ddc5ff64ad79b9f01846a20e2fd at {username='slave'} at 127.28.17.7:38303: session id = c9f94ddc5ff64ad79b9f01846a20e2fd-f30705f9c298491dbfb32bdc554647e4
I20240501 14:01:17.026124 2298 tablet_copy_source_session.cc:205] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.027853 10500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30705f9c298491dbfb32bdc554647e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:01:17.032938 4836 consensus_peers.cc:485] T c685294d3fca4677bc1b1cf355dac3ce P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:17.033630 10500 tablet_copy_client.cc:783] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.033846 10500 tablet_copy_client.cc:647] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.035530 10500 tablet_copy_client.cc:523] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.038929 10500 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap starting.
I20240501 14:01:17.043718 9974 raft_consensus.cc:1050] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:17.044425 9974 consensus_queue.cc:237] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:17.048072 10500 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.048159 5162 raft_consensus.cc:1261] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:17.048465 10500 tablet_bootstrap.cc:492] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Bootstrap complete.
I20240501 14:01:17.048699 10500 ts_tablet_manager.cc:1367] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 14:01:17.048790 10420 consensus_queue.cc:1035] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:17.049403 4836 consensus_peers.cc:485] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.049738 10500 raft_consensus.cc:348] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:17.050006 10500 raft_consensus.cc:729] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Initialized, Role: LEARNER
I20240501 14:01:17.050154 2536 raft_consensus.cc:1261] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:17.050276 10500 consensus_queue.cc:260] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: NON_VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: true } }
I20240501 14:01:17.050824 10420 consensus_queue.cc:1035] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:17.051067 10500 ts_tablet_manager.cc:1398] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:17.051842 2298 tablet_copy_service.cc:342] P b3da1f36f37e4a1aa9f280c2fd0e1fba: Request end of tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-f30705f9c298491dbfb32bdc554647e4 received from {username='slave'} at 127.28.17.7:38303
I20240501 14:01:17.051950 2298 tablet_copy_service.cc:433] P b3da1f36f37e4a1aa9f280c2fd0e1fba: ending tablet copy session c9f94ddc5ff64ad79b9f01846a20e2fd-f30705f9c298491dbfb32bdc554647e4 on tablet f30705f9c298491dbfb32bdc554647e4 with peer c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:17.055869 10004 raft_consensus.cc:2932] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:17.063949 2005 catalog_manager.cc:5484] T 8e605c78442442dd82c598bac2fd84cd P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:17.064281 5162 raft_consensus.cc:2932] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:17.065071 2533 raft_consensus.cc:2932] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:17.090039 10530 ts_tablet_manager.cc:902] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:17.091233 10530 tablet_copy_client.cc:308] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:17.091923 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet e4a0a92e0d48493fac254f1b249a63cf from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:41611)
I20240501 14:01:17.092098 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet e4a0a92e0d48493fac254f1b249a63cf from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:41611: session id = 5a4b9a96e9734baa8a03d6056bfa934c-e4a0a92e0d48493fac254f1b249a63cf
I20240501 14:01:17.095296 2558 tablet_copy_source_session.cc:205] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.097728 10530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a0a92e0d48493fac254f1b249a63cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.104930 10530 tablet_copy_client.cc:783] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.105281 10530 tablet_copy_client.cc:647] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.106689 10190 raft_consensus.cc:1050] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:17.107481 10530 tablet_copy_client.cc:523] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.107491 10190 consensus_queue.cc:237] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:17.112680 10530 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
I20240501 14:01:17.112885 4899 raft_consensus.cc:1261] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:17.113188 2277 raft_consensus.cc:1261] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:17.113896 2407 raft_consensus.cc:1261] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:17.116860 10378 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:17.117509 10378 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:17.117995 10378 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:17.121076 10530 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.121362 10530 tablet_bootstrap.cc:492] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:17.121560 10530 ts_tablet_manager.cc:1367] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240501 14:01:17.122066 10530 raft_consensus.cc:348] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:17.122381 10530 raft_consensus.cc:729] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:01:17.122637 10530 consensus_queue.cc:260] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:17.123080 10530 ts_tablet_manager.cc:1398] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 14:01:17.124953 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-e4a0a92e0d48493fac254f1b249a63cf received from {username='slave'} at 127.28.17.8:41611
I20240501 14:01:17.125113 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-e4a0a92e0d48493fac254f1b249a63cf on tablet e4a0a92e0d48493fac254f1b249a63cf with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:01:17.138435 10536 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:17.139261 2274 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:17.144191 4899 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:17.146672 2405 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:17.148842 2000 catalog_manager.cc:5484] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:17.159062 5293 consensus_queue.cc:237] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:17.163746 2277 raft_consensus.cc:1261] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:17.164358 10190 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:17.164976 2406 raft_consensus.cc:1261] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:17.165603 10190 consensus_queue.cc:1035] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:17.168169 10378 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:17.168632 2277 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:17.175009 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c88cc21c0e0743999ee493f5ee856aef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:17.174893 2406 raft_consensus.cc:2932] T c88cc21c0e0743999ee493f5ee856aef P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:17.176378 2004 catalog_manager.cc:5484] T c88cc21c0e0743999ee493f5ee856aef P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
W20240501 14:01:17.183141 5230 consensus_peers.cc:485] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:17.183993 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet c88cc21c0e0743999ee493f5ee856aef with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:17.188859 10546 tablet_replica.cc:330] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:17.189126 10546 raft_consensus.cc:2229] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:17.189338 10546 raft_consensus.cc:2258] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:17.190197 10546 ts_tablet_manager.cc:1875] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:17.197142 10546 ts_tablet_manager.cc:1888] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:17.197371 10546 log.cc:1196] T c88cc21c0e0743999ee493f5ee856aef P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/c88cc21c0e0743999ee493f5ee856aef
I20240501 14:01:17.198050 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet c88cc21c0e0743999ee493f5ee856aef (table rebalancer_extra_table_12 [id=f11e3fdd692f4f33b496584b72d0ae65]) successfully deleted
I20240501 14:01:17.244273 10573 ts_tablet_manager.cc:902] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: Initiating tablet copy from peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:17.246212 10573 tablet_copy_client.cc:308] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.5:40059
I20240501 14:01:17.246917 4921 tablet_copy_service.cc:140] P f8c60f2bbac541dcba4424ab726f9ad9: Received BeginTabletCopySession request for tablet 9a74335efeb04521baf1720a14093643 from peer 745ac12076e84effa5bde1ecbd6a32ec ({username='slave'} at 127.28.17.4:39663)
I20240501 14:01:17.247068 4921 tablet_copy_service.cc:161] P f8c60f2bbac541dcba4424ab726f9ad9: Beginning new tablet copy session on tablet 9a74335efeb04521baf1720a14093643 from peer 745ac12076e84effa5bde1ecbd6a32ec at {username='slave'} at 127.28.17.4:39663: session id = 745ac12076e84effa5bde1ecbd6a32ec-9a74335efeb04521baf1720a14093643
I20240501 14:01:17.249716 4921 tablet_copy_source_session.cc:205] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.251395 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a74335efeb04521baf1720a14093643. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:01:17.256453 2473 consensus_peers.cc:485] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:17.258247 10573 tablet_copy_client.cc:783] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.258564 10573 tablet_copy_client.cc:647] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.267341 10573 tablet_copy_client.cc:523] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.270900 10573 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap starting.
I20240501 14:01:17.281163 10573 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.281571 10573 tablet_bootstrap.cc:492] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: Bootstrap complete.
I20240501 14:01:17.281888 10573 ts_tablet_manager.cc:1367] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 14:01:17.282718 10573 raft_consensus.cc:348] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:17.283004 10573 raft_consensus.cc:729] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Initialized, Role: LEARNER
I20240501 14:01:17.283221 10573 consensus_queue.cc:260] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: NON_VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: true } }
I20240501 14:01:17.283845 10573 ts_tablet_manager.cc:1398] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:17.285257 4921 tablet_copy_service.cc:342] P f8c60f2bbac541dcba4424ab726f9ad9: Request end of tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-9a74335efeb04521baf1720a14093643 received from {username='slave'} at 127.28.17.4:39663
I20240501 14:01:17.285372 4921 tablet_copy_service.cc:433] P f8c60f2bbac541dcba4424ab726f9ad9: ending tablet copy session 745ac12076e84effa5bde1ecbd6a32ec-9a74335efeb04521baf1720a14093643 on tablet 9a74335efeb04521baf1720a14093643 with peer 745ac12076e84effa5bde1ecbd6a32ec
I20240501 14:01:17.298429 10591 ts_tablet_manager.cc:902] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:17.300444 10591 tablet_copy_client.cc:308] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.4:36117
I20240501 14:01:17.301203 2558 tablet_copy_service.cc:140] P 745ac12076e84effa5bde1ecbd6a32ec: Received BeginTabletCopySession request for tablet 205af5386f904e70989e647da22750e8 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:50935)
I20240501 14:01:17.301369 2558 tablet_copy_service.cc:161] P 745ac12076e84effa5bde1ecbd6a32ec: Beginning new tablet copy session on tablet 205af5386f904e70989e647da22750e8 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:50935: session id = f8c60f2bbac541dcba4424ab726f9ad9-205af5386f904e70989e647da22750e8
I20240501 14:01:17.303803 2558 tablet_copy_source_session.cc:205] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.306094 10591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205af5386f904e70989e647da22750e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.312445 10591 tablet_copy_client.cc:783] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.312736 10591 tablet_copy_client.cc:647] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.314656 10591 tablet_copy_client.cc:523] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.318185 10591 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:17.320067 10530 ts_tablet_manager.cc:902] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: Initiating tablet copy from peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:17.333066 5291 consensus_queue.cc:237] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:17.344269 2406 raft_consensus.cc:1261] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:17.344345 4899 raft_consensus.cc:1261] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:01:17.344775 5228 consensus_peers.cc:485] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d4edc28f28641d4a3201cd896d97afe. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:17.344834 10530 tablet_copy_client.cc:308] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.3:33537
I20240501 14:01:17.344817 10536 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.345052 10378 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:17.345402 2428 tablet_copy_service.cc:140] P a64f34785a7b44fa8070b0452a4e0ddd: Received BeginTabletCopySession request for tablet 67da9784350e49209e979e232a79cfc1 from peer 5a4b9a96e9734baa8a03d6056bfa934c ({username='slave'} at 127.28.17.8:53689)
I20240501 14:01:17.346702 2428 tablet_copy_service.cc:161] P a64f34785a7b44fa8070b0452a4e0ddd: Beginning new tablet copy session on tablet 67da9784350e49209e979e232a79cfc1 from peer 5a4b9a96e9734baa8a03d6056bfa934c at {username='slave'} at 127.28.17.8:53689: session id = 5a4b9a96e9734baa8a03d6056bfa934c-67da9784350e49209e979e232a79cfc1
I20240501 14:01:17.349233 2428 tablet_copy_source_session.cc:205] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.350919 10591 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.351380 10591 tablet_bootstrap.cc:492] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:01:17.351660 10190 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:17.354947 10591 ts_tablet_manager.cc:1367] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.012s
I20240501 14:01:17.355660 10591 raft_consensus.cc:348] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:17.355867 10591 raft_consensus.cc:729] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:01:17.356036 10591 consensus_queue.cc:260] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:17.356654 10591 ts_tablet_manager.cc:1398] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:17.357945 2004 catalog_manager.cc:5484] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:17.358567 10530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67da9784350e49209e979e232a79cfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:17.360811 2558 tablet_copy_service.cc:342] P 745ac12076e84effa5bde1ecbd6a32ec: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-205af5386f904e70989e647da22750e8 received from {username='slave'} at 127.28.17.5:50935
I20240501 14:01:17.360958 2558 tablet_copy_service.cc:433] P 745ac12076e84effa5bde1ecbd6a32ec: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-205af5386f904e70989e647da22750e8 on tablet 205af5386f904e70989e647da22750e8 with peer f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:17.361828 2406 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
I20240501 14:01:17.368912 4899 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } } }
W20240501 14:01:17.381445 4836 consensus_peers.cc:485] T eb9790fa13fe4cb286f0fb7f730da336 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:17.382658 10530 tablet_copy_client.cc:783] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.382913 10530 tablet_copy_client.cc:647] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.386394 10530 tablet_copy_client.cc:523] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.390545 10530 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap starting.
W20240501 14:01:17.397403 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:17.405306 10530 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.405666 10530 tablet_bootstrap.cc:492] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: Bootstrap complete.
I20240501 14:01:17.405894 10530 ts_tablet_manager.cc:1367] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240501 14:01:17.406591 10530 raft_consensus.cc:348] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:17.406813 10530 raft_consensus.cc:729] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Initialized, Role: LEARNER
I20240501 14:01:17.407027 10530 consensus_queue.cc:260] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: NON_VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: true } }
I20240501 14:01:17.407558 10530 ts_tablet_manager.cc:1398] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:17.408362 2428 tablet_copy_service.cc:342] P a64f34785a7b44fa8070b0452a4e0ddd: Request end of tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-67da9784350e49209e979e232a79cfc1 received from {username='slave'} at 127.28.17.8:53689
I20240501 14:01:17.408470 2428 tablet_copy_service.cc:433] P a64f34785a7b44fa8070b0452a4e0ddd: ending tablet copy session 5a4b9a96e9734baa8a03d6056bfa934c-67da9784350e49209e979e232a79cfc1 on tablet 67da9784350e49209e979e232a79cfc1 with peer 5a4b9a96e9734baa8a03d6056bfa934c
I20240501 14:01:17.486261 5291 raft_consensus.cc:1203] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:17.500097 10632 ts_tablet_manager.cc:902] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:17.517864 10632 tablet_copy_client.cc:308] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:01:17.519090 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet 6904a974640a4e84a80b1fb0749365b7 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35917)
I20240501 14:01:17.519259 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet 6904a974640a4e84a80b1fb0749365b7 from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35917: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-6904a974640a4e84a80b1fb0749365b7
I20240501 14:01:17.521844 5316 tablet_copy_source_session.cc:205] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:17.524830 10632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6904a974640a4e84a80b1fb0749365b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:01:17.533047 4836 consensus_peers.cc:485] T ea342bfcde3b4daa8dcc39126fd42cdc P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:17.537457 10632 tablet_copy_client.cc:783] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:17.537721 10632 tablet_copy_client.cc:647] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:17.539469 10632 tablet_copy_client.cc:523] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:17.543526 10632 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:17.545490 5162 raft_consensus.cc:1203] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:01:17.554968 2082 consensus_peers.cc:485] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:17.555949 2213 consensus_peers.cc:485] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:17.570897 10632 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:17.571527 10632 tablet_bootstrap.cc:492] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:17.571874 10632 ts_tablet_manager.cc:1367] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.010s
I20240501 14:01:17.572623 10632 raft_consensus.cc:348] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:17.572942 10632 raft_consensus.cc:729] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:17.573144 10632 consensus_queue.cc:260] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:17.574000 10632 ts_tablet_manager.cc:1398] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:17.574735 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-6904a974640a4e84a80b1fb0749365b7 received from {username='slave'} at 127.28.17.2:35917
I20240501 14:01:17.574851 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-6904a974640a4e84a80b1fb0749365b7 on tablet 6904a974640a4e84a80b1fb0749365b7 with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
W20240501 14:01:17.586596 2082 consensus_peers.cc:485] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:17.615244 2343 consensus_peers.cc:485] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:17.669212 4836 consensus_peers.cc:485] T 926fd4668e8644969365d57749be3e1f P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:17.672276 2343 consensus_peers.cc:485] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:17.709663 2473 consensus_peers.cc:485] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:17.768378 4836 consensus_peers.cc:485] T f647f3f27faf435f81e076bfc2ef0795 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:17.842866 2536 raft_consensus.cc:1203] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:17.882580 5291 raft_consensus.cc:1203] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240501 14:01:17.904211 10632 ts_tablet_manager.cc:902] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: Initiating tablet copy from peer 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:17.904737 4899 raft_consensus.cc:1203] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:17.926528 10228 raft_consensus.cc:1050] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:17.927407 10228 consensus_queue.cc:237] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:17.931622 5291 raft_consensus.cc:1261] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:17.932288 10231 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:17.935820 2274 raft_consensus.cc:1261] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:17.935874 2145 raft_consensus.cc:1261] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:17.937301 10231 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:17.937887 10231 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:01:17.992944 4836 consensus_peers.cc:485] T 83392c08bf314fcd98833335c247ccb1 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.007661 2277 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
W20240501 14:01:18.014550 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:18.006191 10234 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
W20240501 14:01:18.019641 2343 consensus_peers.cc:485] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.040300 2276 raft_consensus.cc:1203] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:01:18.043354 2005 catalog_manager.cc:5484] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:18.056505 2143 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:18.060482 10632 tablet_copy_client.cc:308] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.8:34221
I20240501 14:01:18.060554 5293 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:18.061493 5316 tablet_copy_service.cc:140] P 5a4b9a96e9734baa8a03d6056bfa934c: Received BeginTabletCopySession request for tablet 0d4edc28f28641d4a3201cd896d97afe from peer b3da1f36f37e4a1aa9f280c2fd0e1fba ({username='slave'} at 127.28.17.2:35917)
I20240501 14:01:18.061636 5316 tablet_copy_service.cc:161] P 5a4b9a96e9734baa8a03d6056bfa934c: Beginning new tablet copy session on tablet 0d4edc28f28641d4a3201cd896d97afe from peer b3da1f36f37e4a1aa9f280c2fd0e1fba at {username='slave'} at 127.28.17.2:35917: session id = b3da1f36f37e4a1aa9f280c2fd0e1fba-0d4edc28f28641d4a3201cd896d97afe
I20240501 14:01:18.064455 5316 tablet_copy_source_session.cc:205] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:01:18.065721 1992 catalog_manager.cc:4573] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b9ccb0f16d8044da9f3f353ffbea5ce9: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:18.068475 10632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d4edc28f28641d4a3201cd896d97afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:18.076700 10632 tablet_copy_client.cc:783] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:18.077000 10632 tablet_copy_client.cc:647] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:18.079592 2536 consensus_queue.cc:237] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:18.082468 2274 raft_consensus.cc:1261] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.083055 10700 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.084242 10632 tablet_copy_client.cc:523] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:18.086082 10700 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:18.086755 5293 raft_consensus.cc:1261] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.087390 2274 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
W20240501 14:01:18.092880 2082 consensus_peers.cc:485] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.094579 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4a0a92e0d48493fac254f1b249a63cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:18.094713 10640 raft_consensus.cc:1050] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba: attempting to promote NON_VOTER c9f94ddc5ff64ad79b9f01846a20e2fd to VOTER
I20240501 14:01:18.095697 10640 consensus_queue.cc:237] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:18.097141 10632 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap starting.
I20240501 14:01:18.105412 2004 catalog_manager.cc:5484] T e4a0a92e0d48493fac254f1b249a63cf P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:18.106484 5164 raft_consensus.cc:1261] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 LEARNER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.107378 5293 raft_consensus.cc:1261] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.107512 10308 consensus_queue.cc:1035] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.110285 10234 consensus_queue.cc:1035] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.111817 5291 raft_consensus.cc:2932] T e4a0a92e0d48493fac254f1b249a63cf P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1a414d381f44829a2510cb76d481887 (127.28.17.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
W20240501 14:01:18.116529 2213 consensus_peers.cc:485] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.121208 10640 consensus_queue.cc:1035] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.128861 10632 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:18.130712 10632 tablet_bootstrap.cc:492] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: Bootstrap complete.
I20240501 14:01:18.131012 10632 ts_tablet_manager.cc:1367] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.005s
I20240501 14:01:18.131970 10632 raft_consensus.cc:348] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:18.132468 10632 raft_consensus.cc:729] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Initialized, Role: LEARNER
I20240501 14:01:18.132685 10632 consensus_queue.cc:260] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: NON_VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: true } }
I20240501 14:01:18.133286 10632 ts_tablet_manager.cc:1398] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20240501 14:01:18.137988 5230 consensus_peers.cc:485] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.138795 2125 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a0a92e0d48493fac254f1b249a63cf with delete_type TABLET_DATA_TOMBSTONED (TS d1a414d381f44829a2510cb76d481887 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48706
I20240501 14:01:18.141350 5316 tablet_copy_service.cc:342] P 5a4b9a96e9734baa8a03d6056bfa934c: Request end of tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-0d4edc28f28641d4a3201cd896d97afe received from {username='slave'} at 127.28.17.2:35917
I20240501 14:01:18.141469 5316 tablet_copy_service.cc:433] P 5a4b9a96e9734baa8a03d6056bfa934c: ending tablet copy session b3da1f36f37e4a1aa9f280c2fd0e1fba-0d4edc28f28641d4a3201cd896d97afe on tablet 0d4edc28f28641d4a3201cd896d97afe with peer b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:18.152992 10706 raft_consensus.cc:2932] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:18.153565 5293 raft_consensus.cc:2932] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:18.161702 2000 catalog_manager.cc:5484] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:18.164963 5164 raft_consensus.cc:2932] T f30705f9c298491dbfb32bdc554647e4 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } }
I20240501 14:01:18.173475 10711 tablet_replica.cc:330] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: stopping tablet replica
I20240501 14:01:18.173766 10711 raft_consensus.cc:2229] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:18.174121 10711 raft_consensus.cc:2258] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:18.175410 10711 ts_tablet_manager.cc:1875] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:01:18.179400 2343 consensus_peers.cc:485] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.188522 10711 ts_tablet_manager.cc:1888] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:18.188714 10711 log.cc:1196] T e4a0a92e0d48493fac254f1b249a63cf P d1a414d381f44829a2510cb76d481887: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4a0a92e0d48493fac254f1b249a63cf
I20240501 14:01:18.189527 1990 catalog_manager.cc:4830] TS d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): tablet e4a0a92e0d48493fac254f1b249a63cf (table rebalancer_extra_table_50 [id=446caec1daa74efcb72e594a67b91f1f]) successfully deleted
W20240501 14:01:18.308357 2213 consensus_peers.cc:485] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.317222 5291 consensus_queue.cc:237] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:18.323984 2406 raft_consensus.cc:1261] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:18.324066 2533 raft_consensus.cc:1261] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:18.324558 10488 consensus_queue.cc:1035] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:18.325693 5230 consensus_peers.cc:485] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.325712 10488 consensus_queue.cc:1035] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:18.329026 10138 raft_consensus.cc:1050] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: attempting to promote NON_VOTER 5a4b9a96e9734baa8a03d6056bfa934c to VOTER
I20240501 14:01:18.329815 10138 consensus_queue.cc:237] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:18.329994 10778 raft_consensus.cc:2932] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:18.330617 2406 raft_consensus.cc:2932] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:18.332468 2274 raft_consensus.cc:1261] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:18.332470 5293 raft_consensus.cc:1261] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 LEARNER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:18.332522 2533 raft_consensus.cc:2932] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:18.334059 10147 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.335088 10147 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:18.337502 2082 consensus_peers.cc:485] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.338665 2343 consensus_peers.cc:485] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.342900 2005 catalog_manager.cc:5484] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index -1 to 2, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:18.356529 10470 raft_consensus.cc:2932] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:18.357021 2277 raft_consensus.cc:2932] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
I20240501 14:01:18.365725 2005 catalog_manager.cc:5484] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
W20240501 14:01:18.370041 2343 consensus_peers.cc:485] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.372032 5291 raft_consensus.cc:2932] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } }
W20240501 14:01:18.384763 5099 consensus_peers.cc:485] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:18.401072 2213 consensus_peers.cc:485] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.405064 2213 consensus_peers.cc:485] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.407522 5099 consensus_peers.cc:485] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.411742 9974 raft_consensus.cc:1050] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9: attempting to promote NON_VOTER 745ac12076e84effa5bde1ecbd6a32ec to VOTER
I20240501 14:01:18.412425 9974 consensus_queue.cc:237] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:18.415160 2536 raft_consensus.cc:1261] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEARNER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.415434 2145 raft_consensus.cc:1261] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.416128 10703 consensus_queue.cc:1035] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:01:18.416541 4836 consensus_peers.cc:485] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.416836 10703 consensus_queue.cc:1035] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.419421 10682 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:18.419888 2536 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:18.424711 2145 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:18.425455 2000 catalog_manager.cc:5484] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 2 to 3, 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:18.428469 2274 raft_consensus.cc:1203] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 14:01:18.434626 2213 consensus_peers.cc:485] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.434834 5099 consensus_peers.cc:485] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.442307 4899 consensus_queue.cc:237] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
W20240501 14:01:18.447304 2473 consensus_peers.cc:485] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.448108 2143 raft_consensus.cc:1261] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.448253 2536 raft_consensus.cc:1261] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer f8c60f2bbac541dcba4424ab726f9ad9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.449180 10703 consensus_queue.cc:1035] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.450119 9974 consensus_queue.cc:1035] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.452193 10682 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:18.452629 2533 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:18.457841 2143 raft_consensus.cc:2932] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } } }
I20240501 14:01:18.462595 1991 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a74335efeb04521baf1720a14093643 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:18.464118 2006 catalog_manager.cc:5484] T 9a74335efeb04521baf1720a14093643 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 3 to 4, VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) evicted. New cstate: current_term: 1 leader_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:18.467677 10234 raft_consensus.cc:1050] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec: attempting to promote NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 to VOTER
I20240501 14:01:18.468518 10234 consensus_queue.cc:237] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
W20240501 14:01:18.481528 2473 consensus_peers.cc:485] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.487494 5164 raft_consensus.cc:1261] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.488090 2407 raft_consensus.cc:1261] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.488693 10700 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.488968 10228 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.491724 4899 raft_consensus.cc:1261] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.492434 10700 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 14:01:18.499361 2082 consensus_peers.cc:485] T ce213e03db7f4f17913aad989c584607 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.503700 5230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:01:18.504711 5230 consensus_peers.cc:485] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.513931 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:18.523612 2213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 14:01:18.523200 10700 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
W20240501 14:01:18.524617 2213 consensus_peers.cc:485] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.526875 4901 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
W20240501 14:01:18.530589 2343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 14:01:18.531589 2343 consensus_peers.cc:485] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.532275 5162 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:18.535854 2407 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:18.536743 2006 catalog_manager.cc:5484] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 reported cstate change: config changed from index 2 to 3, f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:18.551255 2533 consensus_queue.cc:237] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:18.554033 4901 raft_consensus.cc:1261] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.554047 2407 raft_consensus.cc:1261] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.554677 10234 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.558738 10700 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:18.561120 10228 consensus_queue.cc:1035] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:18.561786 4836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 114 similar messages]
I20240501 14:01:18.563809 1990 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 205af5386f904e70989e647da22750e8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:01:18.563833 4836 consensus_peers.cc:485] T 86abd8c2e86044f59c8074aa9a77db0d P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.564893 4901 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:18.565280 2406 raft_consensus.cc:2932] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
I20240501 14:01:18.571980 2006 catalog_manager.cc:5484] T 205af5386f904e70989e647da22750e8 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 3 to 4, VOTER c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7) evicted. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } }
W20240501 14:01:18.576977 2082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 94 similar messages]
I20240501 14:01:18.579319 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 205af5386f904e70989e647da22750e8 with delete_type TABLET_DATA_TOMBSTONED (TS c9f94ddc5ff64ad79b9f01846a20e2fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47840
I20240501 14:01:18.580240 10814 tablet_replica.cc:330] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: stopping tablet replica
I20240501 14:01:18.580588 10814 raft_consensus.cc:2229] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:18.582818 10814 raft_consensus.cc:2258] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:18.585337 10814 ts_tablet_manager.cc:1875] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:01:18.592438 2473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240501 14:01:18.595698 10778 raft_consensus.cc:1050] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:18.596505 10778 consensus_queue.cc:237] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:18.598903 4901 raft_consensus.cc:1261] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.599172 2274 raft_consensus.cc:1261] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.599378 10778 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.599749 2407 raft_consensus.cc:1261] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:18.601007 10488 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.601511 10488 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:18.606078 10488 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:18.606818 10814 ts_tablet_manager.cc:1888] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:18.606936 10814 log.cc:1196] T 205af5386f904e70989e647da22750e8 P c9f94ddc5ff64ad79b9f01846a20e2fd: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/wals/205af5386f904e70989e647da22750e8
I20240501 14:01:18.606801 2406 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:18.611230 2274 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:18.614178 2006 catalog_manager.cc:5484] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
W20240501 14:01:18.623382 2473 consensus_peers.cc:485] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.623711 4901 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:18.624402 5293 consensus_queue.cc:237] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:18.627208 1990 catalog_manager.cc:4830] TS c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): tablet 205af5386f904e70989e647da22750e8 (table rebalancer_extra_table_38 [id=579431d2fdc34c2a8bf86a04814411f2]) successfully deleted
I20240501 14:01:18.628086 2274 raft_consensus.cc:1261] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.630626 10825 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:18.630813 2406 raft_consensus.cc:1261] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:18.631497 10825 consensus_queue.cc:1035] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:01:18.631821 2082 consensus_peers.cc:485] T 20cd7aaca5434c2dbb63e7111feed3a9 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.632843 2343 consensus_peers.cc:485] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:18.634397 10825 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:18.634917 2274 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:18.638491 2407 raft_consensus.cc:2932] T 6904a974640a4e84a80b1fb0749365b7 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:18.639989 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6904a974640a4e84a80b1fb0749365b7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:18.641943 2001 catalog_manager.cc:5484] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:18.650894 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 6904a974640a4e84a80b1fb0749365b7 with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:18.653533 10827 tablet_replica.cc:330] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:18.653800 10827 raft_consensus.cc:2229] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:18.654049 10827 raft_consensus.cc:2258] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:18.660889 10827 ts_tablet_manager.cc:1875] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:18.670375 10827 ts_tablet_manager.cc:1888] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:18.670558 10827 log.cc:1196] T 6904a974640a4e84a80b1fb0749365b7 P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/6904a974640a4e84a80b1fb0749365b7
I20240501 14:01:18.671342 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 6904a974640a4e84a80b1fb0749365b7 (table rebalancer_extra_table_10 [id=3cc3a854d12b44bc8f4c2a13e9f0ca88]) successfully deleted
W20240501 14:01:18.683574 4836 consensus_peers.cc:485] T 1fa9e01283eb491abde0417ffbaff56f P f8c60f2bbac541dcba4424ab726f9ad9 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.717834 5230 consensus_peers.cc:485] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.751597 2473 consensus_peers.cc:485] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:18.773037 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67da9784350e49209e979e232a79cfc1"
dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40912
I20240501 14:01:18.773262 2406 raft_consensus.cc:595] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Received request to transfer leadership
W20240501 14:01:18.776080 5099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 36 similar messages]
I20240501 14:01:18.800704 2406 consensus_queue.cc:237] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:18.802989 5164 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:18.803164 2533 raft_consensus.cc:1261] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:18.803627 10147 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:18.803921 10470 consensus_queue.cc:1035] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:01:18.807009 2343 consensus_peers.cc:485] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:18.807236 10138 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:18.808718 5164 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:18.812068 2533 raft_consensus.cc:2932] T fac8d3842d994279b355f9a18a82f901 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
I20240501 14:01:18.815023 2005 catalog_manager.cc:5484] T fac8d3842d994279b355f9a18a82f901 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: config changed from index 6 to 7, NON_VOTER 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6) added. New cstate: current_term: 1 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } } }
W20240501 14:01:18.872228 2082 consensus_peers.cc:485] T 387698a371cc42eab970f2f4471bbf5b P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.887181 2082 consensus_peers.cc:485] T a56d8b76fc244a86b73a741a937bd1c6 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:18.949688 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:18.993399 2406 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67da9784350e49209e979e232a79cfc1"
dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37984
I20240501 14:01:18.993638 2406 raft_consensus.cc:595] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Received request to transfer leadership
I20240501 14:01:19.022677 2143 consensus_queue.cc:237] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:19.025389 5164 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:01:19.026154 2081 consensus_peers.cc:485] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2efec972a06b48b99eb89d454fa214a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:19.026257 2406 raft_consensus.cc:1261] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer d1a414d381f44829a2510cb76d481887: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:19.026616 10946 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:19.027241 10783 consensus_queue.cc:1035] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:19.032037 9788 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:19.032778 5164 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:19.033124 2406 raft_consensus.cc:2932] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:19.034528 10536 raft_consensus.cc:1050] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c: attempting to promote NON_VOTER b3da1f36f37e4a1aa9f280c2fd0e1fba to VOTER
I20240501 14:01:19.035333 10536 consensus_queue.cc:237] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:19.042764 2005 catalog_manager.cc:5484] T 2efec972a06b48b99eb89d454fa214a3 P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:19.045467 2274 raft_consensus.cc:1261] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 LEARNER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:19.046020 4899 raft_consensus.cc:1261] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:19.046986 10825 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:19.053927 2406 raft_consensus.cc:1261] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:01:19.056485 10778 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:19.057111 10778 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:19.072827 10778 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.078102 2274 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.081153 2407 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.084527 2004 catalog_manager.cc:5484] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:19.092096 2533 consensus_queue.cc:237] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:19.095149 5291 consensus_queue.cc:237] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:19.097688 2276 raft_consensus.cc:1261] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:01:19.098789 1992 catalog_manager.cc:4573] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9a74335efeb04521baf1720a14093643: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:19.099083 10778 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:19.101838 2407 raft_consensus.cc:1261] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:01:19.103125 4899 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { replace: true } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.104372 10825 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.107713 10488 consensus_queue.cc:1035] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:19.108577 1992 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d4edc28f28641d4a3201cd896d97afe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:01:19.109207 2472 consensus_peers.cc:485] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8ae4d9e24a94b878627722ad94c0e35. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:19.110157 5291 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:19.111310 2406 raft_consensus.cc:1261] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:01:19.111184 2276 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.111858 10228 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:19.114975 10234 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:19.116338 2407 raft_consensus.cc:2932] T 0d4edc28f28641d4a3201cd896d97afe P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.118929 10700 consensus_queue.cc:1035] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:01:19.118778 2406 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:19.119151 2005 catalog_manager.cc:5484] T 0d4edc28f28641d4a3201cd896d97afe P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: config changed from index 3 to 4, VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) evicted. New cstate: current_term: 1 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } } }
I20240501 14:01:19.127339 2001 catalog_manager.cc:5484] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New cstate: current_term: 1 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:19.127914 4881 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d4edc28f28641d4a3201cd896d97afe with delete_type TABLET_DATA_TOMBSTONED (TS f8c60f2bbac541dcba4424ab726f9ad9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57016
I20240501 14:01:19.132220 10827 tablet_replica.cc:330] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: stopping tablet replica
I20240501 14:01:19.132460 10827 raft_consensus.cc:2229] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:19.132691 10827 raft_consensus.cc:2258] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:19.136199 10827 ts_tablet_manager.cc:1875] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:19.136600 5291 raft_consensus.cc:2932] T f8ae4d9e24a94b878627722ad94c0e35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } } }
I20240501 14:01:19.145488 10827 ts_tablet_manager.cc:1888] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:01:19.145989 10827 log.cc:1196] T 0d4edc28f28641d4a3201cd896d97afe P f8c60f2bbac541dcba4424ab726f9ad9: Deleting WAL directory at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/wals/0d4edc28f28641d4a3201cd896d97afe
I20240501 14:01:19.146821 1991 catalog_manager.cc:4830] TS f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): tablet 0d4edc28f28641d4a3201cd896d97afe (table rebalancer_extra_table_16 [id=8287de96624b49129d31d1c08e179180]) successfully deleted
I20240501 14:01:19.164661 10470 raft_consensus.cc:979] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd: : Instructing follower b3da1f36f37e4a1aa9f280c2fd0e1fba to start an election
I20240501 14:01:19.164881 10782 raft_consensus.cc:1067] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Signalling peer b3da1f36f37e4a1aa9f280c2fd0e1fba to start an election
I20240501 14:01:19.165588 2274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67da9784350e49209e979e232a79cfc1"
dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba"
from {username='slave'} at 127.28.17.3:44257
I20240501 14:01:19.165791 2274 raft_consensus.cc:482] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:19.165880 2274 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:19.168329 2274 raft_consensus.cc:504] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:19.169828 2274 leader_election.cc:288] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 3 election: Requested vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:19.170552 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:19.170764 2407 raft_consensus.cc:3032] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:19.170848 2407 raft_consensus.cc:729] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:01:19.170783 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67da9784350e49209e979e232a79cfc1" candidate_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:19.170943 5291 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:19.171123 2407 consensus_queue.cc:260] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:19.173785 5291 raft_consensus.cc:2454] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 3.
W20240501 14:01:19.178170 2213 leader_election.cc:334] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:19.188162 2407 raft_consensus.cc:3037] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:19.191247 2407 raft_consensus.cc:2454] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3da1f36f37e4a1aa9f280c2fd0e1fba in term 3.
I20240501 14:01:19.191807 2211 leader_election.cc:302] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2
I20240501 14:01:19.192174 10640 raft_consensus.cc:2781] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:19.192337 10640 raft_consensus.cc:686] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 3 LEADER]: Becoming Leader. State: Replica: b3da1f36f37e4a1aa9f280c2fd0e1fba, State: Running, Role: LEADER
I20240501 14:01:19.192624 10640 consensus_queue.cc:237] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:19.197695 2005 catalog_manager.cc:5484] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba reported cstate change: term changed from 2 to 3, leader changed from a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3) to b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2). New cstate: current_term: 3 leader_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:19.239573 5230 consensus_peers.cc:485] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:19.360908 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:19.425693 11062 ts_tablet_manager.cc:902] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:19.437750 11062 tablet_copy_client.cc:308] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.17.1:41309
I20240501 14:01:19.438584 2167 tablet_copy_service.cc:140] P d1a414d381f44829a2510cb76d481887: Received BeginTabletCopySession request for tablet 2efec972a06b48b99eb89d454fa214a3 from peer f8c60f2bbac541dcba4424ab726f9ad9 ({username='slave'} at 127.28.17.5:60335)
I20240501 14:01:19.438761 2167 tablet_copy_service.cc:161] P d1a414d381f44829a2510cb76d481887: Beginning new tablet copy session on tablet 2efec972a06b48b99eb89d454fa214a3 from peer f8c60f2bbac541dcba4424ab726f9ad9 at {username='slave'} at 127.28.17.5:60335: session id = f8c60f2bbac541dcba4424ab726f9ad9-2efec972a06b48b99eb89d454fa214a3
I20240501 14:01:19.441123 2167 tablet_copy_source_session.cc:205] T 2efec972a06b48b99eb89d454fa214a3 P d1a414d381f44829a2510cb76d481887: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:19.446787 11062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efec972a06b48b99eb89d454fa214a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:19.454931 11062 tablet_copy_client.cc:783] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:19.455298 11062 tablet_copy_client.cc:647] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:19.457110 11062 tablet_copy_client.cc:523] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:19.461217 11062 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap starting.
I20240501 14:01:19.535142 11081 ts_tablet_manager.cc:902] T f8ae4d9e24a94b878627722ad94c0e35 P f8c60f2bbac541dcba4424ab726f9ad9: Initiating tablet copy from peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
F20240501 14:01:19.535704 11081 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1cb2dab6f4684bb18bddc9edba1d6517.queue_time_us, tablet-download-pool-1cb2dab6f4684bb18bddc9edba1d6517.queue_time_us) (tablet-download-pool-1cb2dab6f4684bb18bddc9edba1d6517.run_cpu_time_us, tablet-download-pool-1cb2dab6f4684bb18bddc9edba1d6517.run_cpu_time_us) (tablet-download-pool-1cb2dab6f4684bb18bddc9edba1d6517.run_wall_time_us, tablet-download-pool-1cb2dab6f4684bb18bddc9edba1d6517.run_wall_time_us) (tablet-download-pool-205af5386f904e70989e647da22750e8.queue_time_us, tablet-download-pool-205af5386f904e70989e647da22750e8.queue_time_us) (tablet-download-pool-205af5386f904e70989e647da22750e8.run_cpu_time_us, tablet-download-pool-205af5386f904e70989e647da22750e8.run_cpu_time_us) (tablet-download-pool-205af5386f904e70989e647da22750e8.run_wall_time_us, tablet-download-pool-205af5386f904e70989e647da22750e8.run_wall_time_us) (tablet-download-pool-2efec972a06b48b99eb89d454fa214a3.queue_time_us, tablet-download-pool-2efec972a06b48b99eb89d454fa214a3.queue_time_us) (tablet-download-pool-2efec972a06b48b99eb89d454fa214a3.run_cpu_time_us, tablet-download-pool-2efec972a06b48b99eb89d454fa214a3.run_cpu_time_us) (tablet-download-pool-2efec972a06b48b99eb89d454fa214a3.run_wall_time_us, tablet-download-pool-2efec972a06b48b99eb89d454fa214a3.run_wall_time_us) (tablet-download-pool-304b465c0a99456790bfc8f262b61ecd.queue_time_us, tablet-download-pool-304b465c0a99456790bfc8f262b61ecd.queue_time_us) (tablet-download-pool-304b465c0a99456790bfc8f262b61ecd.run_cpu_time_us, tablet-download-pool-304b465c0a99456790bfc8f262b61ecd.run_cpu_time_us) (tablet-download-pool-304b465c0a99456790bfc8f262b61ecd.run_wall_time_us, tablet-download-pool-304b465c0a99456790bfc8f262b61ecd.run_wall_time_us) (tablet-download-pool-44b7e8e96f4e4cc0bd2c824882ac4d84.queue_time_us, tablet-download-pool-44b7e8e96f4e4cc0bd2c824882ac4d84.queue_time_us) (tablet-download-pool-44b7e8e96f4e4cc0bd2c824882ac4d84.run_cpu_time_us, tablet-download-pool-44b7e8e96f4e4cc0bd2c824882ac4d84.run_cpu_time_us) (tablet-download-pool-44b7e8e96f4e4cc0bd2c824882ac4d84.run_wall_time_us, tablet-download-pool-44b7e8e96f4e4cc0bd2c824882ac4d84.run_wall_time_us) (tablet-download-pool-4aa75617adf64020af85f4cf2bd13917.queue_time_us, tablet-download-pool-4aa75617adf64020af85f4cf2bd13917.queue_time_us) (tablet-download-pool-4aa75617adf64020af85f4cf2bd13917.run_cpu_time_us, tablet-download-pool-4aa75617adf64020af85f4cf2bd13917.run_cpu_time_us) (tablet-download-pool-4aa75617adf64020af85f4cf2bd13917.run_wall_time_us, tablet-download-pool-4aa75617adf64020af85f4cf2bd13917.run_wall_time_us) (tablet-download-pool-65306dbef405443fb4c7188ccb762b83.queue_time_us, tablet-download-pool-65306dbef405443fb4c7188ccb762b83.queue_time_us) (tablet-download-pool-65306dbef405443fb4c7188ccb762b83.run_cpu_time_us, tablet-download-pool-65306dbef405443fb4c7188ccb762b83.run_cpu_time_us) (tablet-download-pool-65306dbef405443fb4c7188ccb762b83.run_wall_time_us, tablet-download-pool-65306dbef405443fb4c7188ccb762b83.run_wall_time_us) (tablet-download-pool-6fe1b54933ad446082b1d2840b2c6e49.queue_time_us, tablet-download-pool-6fe1b54933ad446082b1d2840b2c6e49.queue_time_us) (tablet-download-pool-6fe1b54933ad446082b1d2840b2c6e49.run_cpu_time_us, tablet-download-pool-6fe1b54933ad446082b1d2840b2c6e49.run_cpu_time_us) (tablet-download-pool-6fe1b54933ad446082b1d2840b2c6e49.run_wall_time_us, tablet-download-pool-6fe1b54933ad446082b1d2840b2c6e49.run_wall_time_us) (tablet-download-pool-87a79bfca7cd4ffcbb0e4448320d0576.queue_time_us, tablet-download-pool-87a79bfca7cd4ffcbb0e4448320d0576.queue_time_us) (tablet-download-pool-87a79bfca7cd4ffcbb0e4448320d0576.run_cpu_time_us, tablet-download-pool-87a79bfca7cd4ffcbb0e4448320d0576.run_cpu_time_us) (tablet-download-pool-87a79bfca7cd4ffcbb0e4448320d0576.run_wall_time_us, tablet-download-pool-87a79bfca7cd4ffcbb0e4448320d0576.run_wall_time_us) (tablet-download-pool-b3738397b81b46868e09071be6f82334.queue_time_us, tablet-download-pool-b3738397b81b46868e09071be6f82334.queue_time_us) (tablet-download-pool-b3738397b81b46868e09071be6f82334.run_cpu_time_us, tablet-download-pool-b3738397b81b46868e09071be6f82334.run_cpu_time_us) (tablet-download-pool-b3738397b81b46868e09071be6f82334.run_wall_time_us, tablet-download-pool-b3738397b81b46868e09071be6f82334.run_wall_time_us) (tablet-download-pool-eb1fa60c64034b7d8c7b4879823b0d83.queue_time_us, tablet-download-pool-eb1fa60c64034b7d8c7b4879823b0d83.queue_time_us) (tablet-download-pool-eb1fa60c64034b7d8c7b4879823b0d83.run_cpu_time_us, tablet-download-pool-eb1fa60c64034b7d8c7b4879823b0d83.run_cpu_time_us) (tablet-download-pool-eb1fa60c64034b7d8c7b4879823b0d83.run_wall_time_us, tablet-download-pool-eb1fa60c64034b7d8c7b4879823b0d83.run_wall_time_us) (tablet-download-pool-f647f3f27faf435f81e076bfc2ef0795.queue_time_us, tablet-download-pool-f647f3f27faf435f81e076bfc2ef0795.queue_time_us) (tablet-download-pool-f647f3f27faf435f81e076bfc2ef0795.run_cpu_time_us, tablet-download-pool-f647f3f27faf435f81e076bfc2ef0795.run_cpu_time_us) (tablet-download-pool-f647f3f27faf435f81e076bfc2ef0795.run_wall_time_us, tablet-download-pool-f647f3f27faf435f81e076bfc2ef0795.run_wall_time_us) (tablet-download-pool-f8ae4d9e24a94b878627722ad94c0e35.queue_time_us, tablet-download-pool-f8ae4d9e24a94b878627722ad94c0e35.queue_time_us) (tablet-download-pool-fd3f8661a13944098a18d3eea7ddeb00.queue_time_us, tablet-download-pool-fd3f8661a13944098a18d3eea7ddeb00.queue_time_us) (tablet-download-pool-fd3f8661a13944098a18d3eea7ddeb00.run_cpu_time_us, tablet-download-pool-fd3f8661a13944098a18d3eea7ddeb00.run_cpu_time_us) (tablet-download-pool-fd3f8661a13944098a18d3eea7ddeb00.run_wall_time_us, tablet-download-pool-fd3f8661a13944098a18d3eea7ddeb00.run_wall_time_us) (tablet-download-pool-ff4b3345715c44c091ed1073a8a706ff.queue_time_us, tablet-download-pool-ff4b3345715c44c091ed1073a8a706ff.queue_time_us) (tablet-download-pool-ff4b3345715c44c091ed1073a8a706ff.run_cpu_time_us, tablet-download-pool-ff4b3345715c44c091ed1073a8a706ff.run_cpu_time_us) (tablet-download-pool-ff4b3345715c44c091ed1073a8a706ff.run_wall_time_us, tablet-download-pool-ff4b3345715c44c091ed1073a8a706ff.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 14:01:19.541097 11062 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7f5634a12dcd google::LogMessage::Fail() at ??:0
@ 0x7f5634a16b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5634a127cc google::LogMessage::Flush() at ??:0
@ 0x7f5634a13f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5637753742 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:01:19.550923 11062 tablet_bootstrap.cc:492] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: Bootstrap complete.
I20240501 14:01:19.551236 11062 ts_tablet_manager.cc:1367] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent bootstrapping tablet: real 0.090s user 0.048s sys 0.003s
I20240501 14:01:19.552078 11062 raft_consensus.cc:348] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:19.552322 11062 raft_consensus.cc:729] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c60f2bbac541dcba4424ab726f9ad9, State: Initialized, Role: LEARNER
I20240501 14:01:19.552500 11062 consensus_queue.cc:260] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:19.555478 11062 ts_tablet_manager.cc:1398] T 2efec972a06b48b99eb89d454fa214a3 P f8c60f2bbac541dcba4424ab726f9ad9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:19.556370 2167 tablet_copy_service.cc:342] P d1a414d381f44829a2510cb76d481887: Request end of tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-2efec972a06b48b99eb89d454fa214a3 received from {username='slave'} at 127.28.17.5:60335
I20240501 14:01:19.556499 2167 tablet_copy_service.cc:433] P d1a414d381f44829a2510cb76d481887: ending tablet copy session f8c60f2bbac541dcba4424ab726f9ad9-2efec972a06b48b99eb89d454fa214a3 on tablet 2efec972a06b48b99eb89d454fa214a3 with peer f8c60f2bbac541dcba4424ab726f9ad9
@ 0x7f563771ff8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f5637719cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f5646b6c8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f5646b84daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f5646caf480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f5646ce7a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f5646cca981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f5646cca70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5637723b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f563772719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5637727019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f563770201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f563aa876db start_thread at ??:0
@ 0x7f56338c571f clone at ??:0
W20240501 14:01:19.670758 2473 consensus_peers.cc:485] T b147c5f61a574c7d8a334b5f67ce732d P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:19.677511 2213 consensus_peers.cc:485] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.740718 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:19.746369 2407 raft_consensus.cc:1261] T 67da9784350e49209e979e232a79cfc1 P a64f34785a7b44fa8070b0452a4e0ddd [term 3 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:19.746975 10640 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:19.811273 5291 raft_consensus.cc:1261] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 3 FOLLOWER]: Refusing update from remote peer b3da1f36f37e4a1aa9f280c2fd0e1fba: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:19.811959 10640 consensus_queue.cc:1035] T 67da9784350e49209e979e232a79cfc1 P b3da1f36f37e4a1aa9f280c2fd0e1fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:01:19.877774 2081 connection.cc:535] client connection to 127.28.17.5:40059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:19.877856 2472 connection.cc:535] client connection to 127.28.17.5:40059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:19.877879 5098 connection.cc:535] client connection to 127.28.17.5:40059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:19.879412 5229 connection.cc:535] client connection to 127.28.17.5:40059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:19.879489 2212 connection.cc:535] client connection to 127.28.17.5:40059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:19.880124 9915 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:19.878687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.5:40059 (local address 127.28.17.4:55495)
0501 14:01:19.879114 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:19.879132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 14:01:19.879231 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:01:19.879930 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":325}
W20240501 14:01:19.880295 2472 consensus_peers.cc:485] T 1cb2dab6f4684bb18bddc9edba1d6517 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.880378 2472 consensus_peers.cc:462] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:19.880415 2472 consensus_peers.cc:485] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.880461 2472 consensus_peers.cc:485] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.880515 2472 consensus_peers.cc:485] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.880504 5229 consensus_peers.cc:485] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.880586 2472 consensus_peers.cc:485] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.880650 2472 consensus_peers.cc:485] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.880839 9923 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:19.878570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.5:40059 (local address 127.28.17.7:42785)
0501 14:01:19.878726 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:19.878744 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 14:01:19.878848 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:01:19.879948 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35}
W20240501 14:01:19.881013 2342 connection.cc:535] client connection to 127.28.17.5:40059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:19.882153 9922 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:19.878898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.5:40059 (local address 127.28.17.1:41467)
0501 14:01:19.879115 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:19.879132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 14:01:19.879231 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:01:19.881155 (+ 1924us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":106}
W20240501 14:01:19.882638 2212 consensus_peers.cc:485] T 65306dbef405443fb4c7188ccb762b83 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.882663 5098 consensus_peers.cc:485] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.882845 2342 consensus_peers.cc:485] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.882853 2081 consensus_peers.cc:485] T 27abced2b6b746d2b7c45883a328a3d3 P d1a414d381f44829a2510cb76d481887 -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.882952 2342 consensus_peers.cc:485] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.888625 2212 consensus_peers.cc:485] T 9935f0f87637467091ff85989503af28 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.916414 2472 consensus_peers.cc:485] T 205af5386f904e70989e647da22750e8 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.927362 2081 consensus_peers.cc:485] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.945305 2212 consensus_peers.cc:485] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.954102 2212 consensus_peers.cc:485] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.961223 2212 consensus_peers.cc:485] T 91184ecf7ec3409ca3b05318aa5dcab3 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.968493 2212 consensus_peers.cc:485] T b3738397b81b46868e09071be6f82334 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.971185 2343 consensus_peers.cc:485] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:19.989287 5098 consensus_peers.cc:485] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.989621 5229 consensus_peers.cc:485] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:19.995770 5098 consensus_peers.cc:485] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:20.015086 2472 consensus_peers.cc:462] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:20.052598 2082 consensus_peers.cc:485] T 913ae5fda9e842e1b3a6d7dc418a5bfd P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:20.068122 2081 consensus_peers.cc:485] T fd3f8661a13944098a18d3eea7ddeb00 P d1a414d381f44829a2510cb76d481887 -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:20.070931 2081 consensus_peers.cc:485] T 44b7e8e96f4e4cc0bd2c824882ac4d84 P d1a414d381f44829a2510cb76d481887 -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:20.074995 2213 consensus_peers.cc:485] T a7b11568c2d7420fa06f903e52723583 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:20.091850 2342 consensus_peers.cc:485] T a095254d78344a28ba5487f742bcfb4d P a64f34785a7b44fa8070b0452a4e0ddd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:20.115820 2082 consensus_peers.cc:485] T bf051cecef424db5accfce66623346b5 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:20.182969 1992 catalog_manager.cc:4573] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet afc49574262e4819b6c86025c43c772a: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:20.223446 2343 consensus_peers.cc:485] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.271569 5276 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:01:20.276506 2517 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 14:01:20.283963 2473 consensus_peers.cc:485] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.317552 2388 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:01:20.326943 2258 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 14:01:20.357350 5142 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240501 14:01:20.357491 2082 consensus_peers.cc:462] T e3fe9fe1c13743478e09f5eeeb528b57 P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:20.397107 2125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 14:01:20.476312 2082 consensus_peers.cc:485] T 0a489f3c24c54297a0569dd560cf159e P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:20.527714 2213 consensus_peers.cc:485] T f30705f9c298491dbfb32bdc554647e4 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:20.539145 2343 consensus_peers.cc:485] T 6c74bacbf4464bab9c403e2153953474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.556679 10489 raft_consensus.cc:482] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.556867 10489 raft_consensus.cc:504] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:20.557473 10489 leader_election.cc:288] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:20.558921 5099 leader_election.cc:334] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:20.559373 5098 leader_election.cc:334] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.559481 5098 leader_election.cc:302] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.559707 10489 raft_consensus.cc:2726] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:01:20.564471 2472 consensus_peers.cc:462] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:20.586753 2343 consensus_peers.cc:485] T 01096bd6f2bf4663b70043f61ecdea5b P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.602599 10489 raft_consensus.cc:482] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.602785 10489 raft_consensus.cc:504] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:20.603510 10489 leader_election.cc:288] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:20.604312 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:20.604450 10825 raft_consensus.cc:482] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.604609 10825 raft_consensus.cc:504] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
W20240501 14:01:20.605018 5098 leader_election.cc:334] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.605144 5097 leader_election.cc:302] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.605151 10825 leader_election.cc:288] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:20.605412 10489 raft_consensus.cc:2726] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:20.605788 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697989873e94a6fad3835e0ebb36fce" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
W20240501 14:01:20.607072 5229 leader_election.cc:334] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.607208 5229 leader_election.cc:302] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.607429 10825 raft_consensus.cc:2726] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:20.611116 11175 raft_consensus.cc:482] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.611267 11175 raft_consensus.cc:504] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:20.611814 11175 leader_election.cc:288] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:20.612226 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2f1ffd51584255b020ed742f30f573" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
W20240501 14:01:20.613131 2081 leader_election.cc:334] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.613224 2081 leader_election.cc:302] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.613449 11175 raft_consensus.cc:2726] T 1d2f1ffd51584255b020ed742f30f573 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:20.627593 5099 consensus_peers.cc:485] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:20.630584 2213 consensus_peers.cc:485] T 7655e45073d947eabec36e6de7d6a688 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.632877 11175 raft_consensus.cc:482] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.633036 11175 raft_consensus.cc:504] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:20.633697 11175 leader_election.cc:288] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
W20240501 14:01:20.635301 2081 leader_election.cc:334] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.635856 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e20cd9040e4448b93ac513371a7fcf9" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" is_pre_election: true
I20240501 14:01:20.636413 2080 leader_election.cc:302] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.637100 11175 raft_consensus.cc:2726] T 2e20cd9040e4448b93ac513371a7fcf9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:20.699793 10234 raft_consensus.cc:482] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
W20240501 14:01:20.700206 5230 consensus_peers.cc:485] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.700111 10234 raft_consensus.cc:504] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:20.701534 10234 leader_election.cc:288] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:20.702425 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:20.702665 5162 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 2.
W20240501 14:01:20.704027 2472 leader_election.cc:334] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.704461 2471 leader_election.cc:302] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.705003 10234 raft_consensus.cc:2781] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:01:20.705077 10234 raft_consensus.cc:482] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting leader election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.705137 10234 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:20.707926 10234 raft_consensus.cc:504] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:20.708484 10234 leader_election.cc:288] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:20.708956 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eec5cc39eda4f6d85f83959952a322c" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:20.709170 5162 raft_consensus.cc:3037] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:01:20.709743 2472 leader_election.cc:334] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.711380 5162 raft_consensus.cc:2454] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 3.
I20240501 14:01:20.711750 2471 leader_election.cc:302] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.712019 10234 raft_consensus.cc:2781] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:20.712129 10234 raft_consensus.cc:686] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [term 3 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:01:20.712337 10234 consensus_queue.cc:237] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:20.714010 11217 raft_consensus.cc:482] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.714146 11217 raft_consensus.cc:504] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } attrs { promote: false } }
I20240501 14:01:20.714778 11217 leader_election.cc:288] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:20.716428 2213 leader_election.cc:334] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:20.716718 2006 catalog_manager.cc:5484] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 2 to 3, leader changed from f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 3 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:20.717059 2212 leader_election.cc:334] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.717155 2212 leader_election.cc:302] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.717370 11217 raft_consensus.cc:2726] T 1fa9e01283eb491abde0417ffbaff56f P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:20.745664 11219 raft_consensus.cc:482] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.745854 11219 raft_consensus.cc:504] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:20.746749 11219 leader_election.cc:288] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:20.747442 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4e5590d124918ac4814a86b006b69" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:20.747660 2533 raft_consensus.cc:2454] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
W20240501 14:01:20.748188 2081 leader_election.cc:334] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.748533 2080 leader_election.cc:302] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.748800 11219 raft_consensus.cc:2781] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:20.748896 11219 raft_consensus.cc:482] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.748970 11219 raft_consensus.cc:3037] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:20.750989 11219 raft_consensus.cc:504] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:20.751624 11219 leader_election.cc:288] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:20.752099 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4e5590d124918ac4814a86b006b69" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec"
I20240501 14:01:20.752296 2533 raft_consensus.cc:3037] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:20.753175 2081 leader_election.cc:334] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.754377 2533 raft_consensus.cc:2454] T a9d4e5590d124918ac4814a86b006b69 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 2.
I20240501 14:01:20.754912 2080 leader_election.cc:302] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.755196 11219 raft_consensus.cc:2781] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:20.755334 11219 raft_consensus.cc:686] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [term 2 LEADER]: Becoming Leader. State: Replica: d1a414d381f44829a2510cb76d481887, State: Running, Role: LEADER
I20240501 14:01:20.755580 11219 consensus_queue.cc:237] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:20.756384 10489 raft_consensus.cc:482] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.756521 10489 raft_consensus.cc:504] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:20.757409 10489 leader_election.cc:288] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:20.758888 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
W20240501 14:01:20.759042 5099 leader_election.cc:334] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:20.759759 5098 leader_election.cc:334] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.759869 5098 leader_election.cc:302] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.760274 10489 raft_consensus.cc:2726] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:20.760186 2006 catalog_manager.cc:5484] T a9d4e5590d124918ac4814a86b006b69 P d1a414d381f44829a2510cb76d481887 reported cstate change: term changed from 1 to 2, leader changed from f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) to d1a414d381f44829a2510cb76d481887 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "d1a414d381f44829a2510cb76d481887" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:20.772364 11221 raft_consensus.cc:482] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.772542 11221 raft_consensus.cc:504] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:20.773253 11221 leader_election.cc:288] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:20.774636 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaae937ea4d44068d3ad5b9024b9421" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
W20240501 14:01:20.775933 2342 leader_election.cc:334] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.776055 2342 leader_election.cc:302] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.776342 11221 raft_consensus.cc:2726] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:20.785336 10489 raft_consensus.cc:482] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.785499 10489 raft_consensus.cc:504] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:20.786177 10489 leader_election.cc:288] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:20.787566 5098 leader_election.cc:334] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:20.787698 5099 leader_election.cc:334] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:20.787783 5099 leader_election.cc:302] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.788146 10489 raft_consensus.cc:2726] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:01:20.814036 2082 consensus_peers.cc:485] T 0e407965bc6047b185913b10dd0681fa P d1a414d381f44829a2510cb76d481887 -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.815145 11221 raft_consensus.cc:482] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.815603 11221 raft_consensus.cc:504] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } }
I20240501 14:01:20.816254 11221 leader_election.cc:288] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:20.820706 2342 leader_election.cc:334] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.820834 11175 raft_consensus.cc:482] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.820989 11175 raft_consensus.cc:504] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
W20240501 14:01:20.821106 2343 leader_election.cc:334] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:20.821336 2343 leader_election.cc:302] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.821656 11175 leader_election.cc:288] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:20.821826 11221 raft_consensus.cc:2726] T 926fd4668e8644969365d57749be3e1f P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:20.823009 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccb0f16d8044da9f3f353ffbea5ce9" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
W20240501 14:01:20.823485 2081 leader_election.cc:334] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.823671 2080 leader_election.cc:302] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1a414d381f44829a2510cb76d481887; no voters: a64f34785a7b44fa8070b0452a4e0ddd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.823884 11175 raft_consensus.cc:2726] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-task84tCje/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 41
Maximum Replica Count | 52
Average Replica Count | 46.500000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.190476
stderr: W20240501 14:01:04.601440 8724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:04.601789 8724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:04.611173 8724 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:06.031998 8724 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:01:06.782043 8724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a4f7d80ee1547c2a8ccbe684a985094: could not find any suitable replica to move from server 6f81f803ec7746ccb502c59b7cbf8eb2 to server a64f34785a7b44fa8070b0452a4e0ddd
W20240501 14:01:06.782196 8724 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d1861840c97941dc8197bed58b21816f: could not find any suitable replica to move from server 745ac12076e84effa5bde1ecbd6a32ec to server d1a414d381f44829a2510cb76d481887
I20240501 14:01:06.816098 8724 rebalancer_tool.cc:1074] tablet 3bc3084368de4add8809f10451d96894: '745ac12076e84effa5bde1ecbd6a32ec' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:06.900877 8724 rebalancer_tool.cc:1074] tablet f6e71ec5dcbe4a08a0a7da5944dfe77d: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'd1a414d381f44829a2510cb76d481887' move scheduled
I20240501 14:01:06.966450 8724 rebalancer_tool.cc:1074] tablet 9eec5cc39eda4f6d85f83959952a322c: '5a4b9a96e9734baa8a03d6056bfa934c' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move scheduled
I20240501 14:01:06.998358 8724 rebalancer_tool.cc:1074] tablet a7b11568c2d7420fa06f903e52723583: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move scheduled
I20240501 14:01:07.041918 8724 rebalancer_tool.cc:1074] tablet 8cdc95819b944c5590202e86cc955d91: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:07.083716 8724 rebalancer_tool.cc:1074] tablet e8e883d12de74534a02ca9ccaada086f: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:07.111086 8724 rebalancer_tool.cc:1074] tablet 65306dbef405443fb4c7188ccb762b83: 'd1a414d381f44829a2510cb76d481887' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move scheduled
I20240501 14:01:07.164113 8724 rebalancer_tool.cc:1074] tablet 878eb076eee94aa8988b3588d5395cba: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move scheduled
I20240501 14:01:07.206980 8724 rebalancer_tool.cc:1074] tablet f1e048df08cf4f7eaacb78d6592d79b1: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move scheduled
I20240501 14:01:07.235080 8724 rebalancer_tool.cc:1074] tablet 6c74bacbf4464bab9c403e2153953474: '745ac12076e84effa5bde1ecbd6a32ec' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:07.262099 8724 rebalancer_tool.cc:1074] tablet 4aa75617adf64020af85f4cf2bd13917: 'd1a414d381f44829a2510cb76d481887' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move scheduled
I20240501 14:01:07.305685 8724 rebalancer_tool.cc:1074] tablet ea342bfcde3b4daa8dcc39126fd42cdc: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move scheduled
I20240501 14:01:07.359651 8724 rebalancer_tool.cc:1074] tablet 44a4e92b144f47b5800506f4bc3eadb6: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:07.496042 8724 rebalancer_tool.cc:1074] tablet a9d4e5590d124918ac4814a86b006b69: '5a4b9a96e9734baa8a03d6056bfa934c' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:07.610929 8724 rebalancer_tool.cc:1074] tablet b9ce2e195d834f6b986ac7273e7c6b78: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'd1a414d381f44829a2510cb76d481887' move scheduled
I20240501 14:01:07.815773 8724 rebalancer_tool.cc:1074] tablet fd3f8661a13944098a18d3eea7ddeb00: 'a64f34785a7b44fa8070b0452a4e0ddd' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move scheduled
I20240501 14:01:08.006958 8724 rebalancer_tool.cc:1074] tablet e9a9e1d4df0a48f59b03349380cb6608: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:08.330974 8724 rebalancer_tool.cc:1074] tablet 86abd8c2e86044f59c8074aa9a77db0d: '5a4b9a96e9734baa8a03d6056bfa934c' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:09.003933 8724 rebalancer_tool.cc:1074] tablet bf051cecef424db5accfce66623346b5: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'd1a414d381f44829a2510cb76d481887' move scheduled
I20240501 14:01:09.405362 8724 rebalancer_tool.cc:1136] tablet 8cdc95819b944c5590202e86cc955d91: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> '745ac12076e84effa5bde1ecbd6a32ec' move completed: OK
I20240501 14:01:09.480023 8724 rebalancer_tool.cc:1136] tablet a7b11568c2d7420fa06f903e52723583: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move completed: OK
I20240501 14:01:09.605095 8724 rebalancer_tool.cc:1136] tablet f1e048df08cf4f7eaacb78d6592d79b1: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move completed: OK
I20240501 14:01:09.627940 8724 rebalancer_tool.cc:1136] tablet 6c74bacbf4464bab9c403e2153953474: '745ac12076e84effa5bde1ecbd6a32ec' -> '5a4b9a96e9734baa8a03d6056bfa934c' move completed: OK
I20240501 14:01:09.756646 8724 rebalancer_tool.cc:1136] tablet 3bc3084368de4add8809f10451d96894: '745ac12076e84effa5bde1ecbd6a32ec' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:09.778652 8724 rebalancer_tool.cc:1136] tablet e8e883d12de74534a02ca9ccaada086f: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '5a4b9a96e9734baa8a03d6056bfa934c' move completed: OK
I20240501 14:01:09.801111 8724 rebalancer_tool.cc:1136] tablet 878eb076eee94aa8988b3588d5395cba: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move completed: OK
I20240501 14:01:09.815364 8724 rebalancer_tool.cc:1136] tablet f6e71ec5dcbe4a08a0a7da5944dfe77d: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'd1a414d381f44829a2510cb76d481887' move completed: OK
I20240501 14:01:09.829499 8724 rebalancer_tool.cc:1136] tablet 65306dbef405443fb4c7188ccb762b83: 'd1a414d381f44829a2510cb76d481887' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move completed: OK
I20240501 14:01:09.984974 8724 rebalancer_tool.cc:1136] tablet a9d4e5590d124918ac4814a86b006b69: '5a4b9a96e9734baa8a03d6056bfa934c' -> '745ac12076e84effa5bde1ecbd6a32ec' move completed: OK
I20240501 14:01:10.033277 8724 rebalancer_tool.cc:1136] tablet b9ce2e195d834f6b986ac7273e7c6b78: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'd1a414d381f44829a2510cb76d481887' move completed: OK
I20240501 14:01:10.069196 8724 rebalancer_tool.cc:1136] tablet 4aa75617adf64020af85f4cf2bd13917: 'd1a414d381f44829a2510cb76d481887' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move completed: OK
I20240501 14:01:10.080865 8724 rebalancer_tool.cc:1136] tablet e9a9e1d4df0a48f59b03349380cb6608: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:10.135730 8724 rebalancer_tool.cc:1136] tablet ea342bfcde3b4daa8dcc39126fd42cdc: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move completed: OK
I20240501 14:01:10.191774 8724 rebalancer_tool.cc:1136] tablet 44a4e92b144f47b5800506f4bc3eadb6: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:10.235893 8724 rebalancer_tool.cc:1074] tablet 0fac2baeae824f68ae2fe1c62e36d970: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:10.276175 8724 rebalancer_tool.cc:1074] tablet ead586b9a8024a218c55d2bb579a16de: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> 'd1a414d381f44829a2510cb76d481887' move scheduled
I20240501 14:01:10.306814 8724 rebalancer_tool.cc:1074] tablet 4ed332b3f7304bf69bc348d230ece128: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move scheduled
I20240501 14:01:10.340690 8724 rebalancer_tool.cc:1074] tablet 566acdef1d114b8aac9b55ccfa03da35: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:10.378031 8724 rebalancer_tool.cc:1074] tablet 9b91e5a14d82419cb295439f577d2786: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:10.440441 8724 rebalancer_tool.cc:1074] tablet ce213e03db7f4f17913aad989c584607: '745ac12076e84effa5bde1ecbd6a32ec' -> 'd1a414d381f44829a2510cb76d481887' move scheduled
I20240501 14:01:10.483472 8724 rebalancer_tool.cc:1074] tablet 741357494d3a4842935671afd1dbe677: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move scheduled
I20240501 14:01:10.545711 8724 rebalancer_tool.cc:1074] tablet 0e407965bc6047b185913b10dd0681fa: '5a4b9a96e9734baa8a03d6056bfa934c' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move scheduled
I20240501 14:01:10.685935 8724 rebalancer_tool.cc:1074] tablet 0a489f3c24c54297a0569dd560cf159e: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:10.810005 8724 rebalancer_tool.cc:1074] tablet f647f3f27faf435f81e076bfc2ef0795: '745ac12076e84effa5bde1ecbd6a32ec' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move scheduled
I20240501 14:01:10.943075 8724 rebalancer_tool.cc:1074] tablet 1cb2dab6f4684bb18bddc9edba1d6517: 'd1a414d381f44829a2510cb76d481887' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:11.138367 8724 rebalancer_tool.cc:1074] tablet fdb2924e91fb4a98aaa6bf977f682fdb: 'a64f34785a7b44fa8070b0452a4e0ddd' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move scheduled
I20240501 14:01:11.323850 8724 rebalancer_tool.cc:1074] tablet b911b0ff6ef446d8a6637edca57e570d: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:11.403079 8724 rebalancer_tool.cc:1074] tablet beb5db06e4b74430b2bcbbecc82796e2: 'f8c60f2bbac541dcba4424ab726f9ad9' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:11.507861 8724 rebalancer_tool.cc:1074] tablet e3fe9fe1c13743478e09f5eeeb528b57: 'a64f34785a7b44fa8070b0452a4e0ddd' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move scheduled
I20240501 14:01:13.025139 8724 rebalancer_tool.cc:1136] tablet 9b91e5a14d82419cb295439f577d2786: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:13.097806 8724 rebalancer_tool.cc:1136] tablet 566acdef1d114b8aac9b55ccfa03da35: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '5a4b9a96e9734baa8a03d6056bfa934c' move completed: OK
I20240501 14:01:13.191385 8724 rebalancer_tool.cc:1136] tablet 4ed332b3f7304bf69bc348d230ece128: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move completed: OK
I20240501 14:01:13.284561 8724 rebalancer_tool.cc:1136] tablet ead586b9a8024a218c55d2bb579a16de: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> 'd1a414d381f44829a2510cb76d481887' move completed: OK
I20240501 14:01:13.320130 8724 rebalancer_tool.cc:1136] tablet 0fac2baeae824f68ae2fe1c62e36d970: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '745ac12076e84effa5bde1ecbd6a32ec' move completed: OK
I20240501 14:01:13.360136 8724 rebalancer_tool.cc:1136] tablet 1cb2dab6f4684bb18bddc9edba1d6517: 'd1a414d381f44829a2510cb76d481887' -> '5a4b9a96e9734baa8a03d6056bfa934c' move completed: OK
I20240501 14:01:13.412703 8724 rebalancer_tool.cc:1136] tablet 9eec5cc39eda4f6d85f83959952a322c: '5a4b9a96e9734baa8a03d6056bfa934c' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move completed: OK
I20240501 14:01:13.459995 8724 rebalancer_tool.cc:1136] tablet 86abd8c2e86044f59c8074aa9a77db0d: '5a4b9a96e9734baa8a03d6056bfa934c' -> '745ac12076e84effa5bde1ecbd6a32ec' move completed: OK
I20240501 14:01:13.567270 8724 rebalancer_tool.cc:1123] tablet bf051cecef424db5accfce66623346b5: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'd1a414d381f44829a2510cb76d481887' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:13.582397 8724 rebalancer_tool.cc:1136] tablet fd3f8661a13944098a18d3eea7ddeb00: 'a64f34785a7b44fa8070b0452a4e0ddd' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move completed: OK
I20240501 14:01:13.707598 8724 rebalancer_tool.cc:1074] tablet 52eb73b91ed441199eade33e66b84b39: '5a4b9a96e9734baa8a03d6056bfa934c' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:13.760318 8724 rebalancer_tool.cc:1074] tablet 0ff679e55a5446099efbdbcd1109f245: 'd1a414d381f44829a2510cb76d481887' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:13.829692 8724 rebalancer_tool.cc:1074] tablet 5c4312eec255417580288818bd559099: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move scheduled
I20240501 14:01:13.889482 8724 rebalancer_tool.cc:1074] tablet ccec6fa18f844ace900db8bfe91738a5: 'a64f34785a7b44fa8070b0452a4e0ddd' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:13.940428 8724 rebalancer_tool.cc:1074] tablet b9d8d993ce834c1db4dd4418f7a85780: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:13.993067 8724 rebalancer_tool.cc:1074] tablet eb1fa60c64034b7d8c7b4879823b0d83: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'd1a414d381f44829a2510cb76d481887' move scheduled
I20240501 14:01:14.038887 8724 rebalancer_tool.cc:1074] tablet afc49574262e4819b6c86025c43c772a: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move scheduled
I20240501 14:01:14.081931 8724 rebalancer_tool.cc:1074] tablet efbd148f37bb46f0a4bf0fc562fff597: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move scheduled
I20240501 14:01:14.126924 8724 rebalancer_tool.cc:1074] tablet 36f03024767442ac911a8a44f7ecdceb: '5a4b9a96e9734baa8a03d6056bfa934c' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:14.260778 8724 rebalancer_tool.cc:1136] tablet beb5db06e4b74430b2bcbbecc82796e2: 'f8c60f2bbac541dcba4424ab726f9ad9' -> '745ac12076e84effa5bde1ecbd6a32ec' move completed: OK
I20240501 14:01:14.272361 8724 rebalancer_tool.cc:1136] tablet b911b0ff6ef446d8a6637edca57e570d: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:14.286943 8724 rebalancer_tool.cc:1123] tablet f647f3f27faf435f81e076bfc2ef0795: '745ac12076e84effa5bde1ecbd6a32ec' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:14.296633 8724 rebalancer_tool.cc:1136] tablet 0a489f3c24c54297a0569dd560cf159e: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:14.307700 8724 rebalancer_tool.cc:1136] tablet 0e407965bc6047b185913b10dd0681fa: '5a4b9a96e9734baa8a03d6056bfa934c' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move completed: OK
I20240501 14:01:14.317332 8724 rebalancer_tool.cc:1136] tablet fdb2924e91fb4a98aaa6bf977f682fdb: 'a64f34785a7b44fa8070b0452a4e0ddd' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move completed: OK
I20240501 14:01:14.328519 8724 rebalancer_tool.cc:1136] tablet 741357494d3a4842935671afd1dbe677: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move completed: OK
I20240501 14:01:14.491168 8724 rebalancer_tool.cc:1074] tablet 6fe1b54933ad446082b1d2840b2c6e49: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move scheduled
I20240501 14:01:14.896564 8724 rebalancer_tool.cc:1074] tablet e081778a0f67486a904f3bed7dfe8644: '5a4b9a96e9734baa8a03d6056bfa934c' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:15.319769 8724 rebalancer_tool.cc:1074] tablet b9ccb0f16d8044da9f3f353ffbea5ce9: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move scheduled
I20240501 14:01:15.387050 8724 rebalancer_tool.cc:1074] tablet 8e605c78442442dd82c598bac2fd84cd: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:15.534894 8724 rebalancer_tool.cc:1074] tablet c88cc21c0e0743999ee493f5ee856aef: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:16.216517 8724 rebalancer_tool.cc:1136] tablet 6fe1b54933ad446082b1d2840b2c6e49: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move completed: OK
I20240501 14:01:16.296080 8724 rebalancer_tool.cc:1136] tablet eb1fa60c64034b7d8c7b4879823b0d83: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'd1a414d381f44829a2510cb76d481887' move completed: OK
I20240501 14:01:16.308884 8724 rebalancer_tool.cc:1136] tablet b9d8d993ce834c1db4dd4418f7a85780: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> '5a4b9a96e9734baa8a03d6056bfa934c' move completed: OK
I20240501 14:01:16.330446 8724 rebalancer_tool.cc:1136] tablet afc49574262e4819b6c86025c43c772a: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move completed: OK
I20240501 14:01:16.341357 8724 rebalancer_tool.cc:1136] tablet ccec6fa18f844ace900db8bfe91738a5: 'a64f34785a7b44fa8070b0452a4e0ddd' -> '745ac12076e84effa5bde1ecbd6a32ec' move completed: OK
I20240501 14:01:16.355238 8724 rebalancer_tool.cc:1136] tablet 52eb73b91ed441199eade33e66b84b39: '5a4b9a96e9734baa8a03d6056bfa934c' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:16.381726 8724 rebalancer_tool.cc:1136] tablet 0ff679e55a5446099efbdbcd1109f245: 'd1a414d381f44829a2510cb76d481887' -> '5a4b9a96e9734baa8a03d6056bfa934c' move completed: OK
I20240501 14:01:16.471045 8724 rebalancer_tool.cc:1136] tablet 5c4312eec255417580288818bd559099: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move completed: OK
I20240501 14:01:16.583498 8724 rebalancer_tool.cc:1074] tablet f30705f9c298491dbfb32bdc554647e4: '5a4b9a96e9734baa8a03d6056bfa934c' -> 'c9f94ddc5ff64ad79b9f01846a20e2fd' move scheduled
I20240501 14:01:16.622705 8724 rebalancer_tool.cc:1074] tablet e4a0a92e0d48493fac254f1b249a63cf: 'd1a414d381f44829a2510cb76d481887' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:16.684957 8724 rebalancer_tool.cc:1074] tablet 9a74335efeb04521baf1720a14093643: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '745ac12076e84effa5bde1ecbd6a32ec' move scheduled
I20240501 14:01:16.747335 8724 rebalancer_tool.cc:1074] tablet 205af5386f904e70989e647da22750e8: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move scheduled
I20240501 14:01:16.795223 8724 rebalancer_tool.cc:1074] tablet 67da9784350e49209e979e232a79cfc1: 'a64f34785a7b44fa8070b0452a4e0ddd' -> '5a4b9a96e9734baa8a03d6056bfa934c' move scheduled
I20240501 14:01:16.898535 8724 rebalancer_tool.cc:1074] tablet 6904a974640a4e84a80b1fb0749365b7: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:17.188361 8724 rebalancer_tool.cc:1136] tablet c88cc21c0e0743999ee493f5ee856aef: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:17.205404 8724 rebalancer_tool.cc:1136] tablet b9ccb0f16d8044da9f3f353ffbea5ce9: '6f81f803ec7746ccb502c59b7cbf8eb2' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move completed: OK
I20240501 14:01:17.236588 8724 rebalancer_tool.cc:1136] tablet efbd148f37bb46f0a4bf0fc562fff597: 'b3da1f36f37e4a1aa9f280c2fd0e1fba' -> 'a64f34785a7b44fa8070b0452a4e0ddd' move completed: OK
I20240501 14:01:17.357440 8724 rebalancer_tool.cc:1074] tablet 0d4edc28f28641d4a3201cd896d97afe: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move scheduled
I20240501 14:01:18.216885 8724 rebalancer_tool.cc:1136] tablet e4a0a92e0d48493fac254f1b249a63cf: 'd1a414d381f44829a2510cb76d481887' -> '5a4b9a96e9734baa8a03d6056bfa934c' move completed: OK
I20240501 14:01:18.337457 8724 rebalancer_tool.cc:1074] tablet 1b6c6fcfebde46f18c55cf51c6b2f4f2: '5a4b9a96e9734baa8a03d6056bfa934c' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move scheduled
I20240501 14:01:18.587183 8724 rebalancer_tool.cc:1136] tablet 9a74335efeb04521baf1720a14093643: '6f81f803ec7746ccb502c59b7cbf8eb2' -> '745ac12076e84effa5bde1ecbd6a32ec' move completed: OK
I20240501 14:01:18.826725 8724 rebalancer_tool.cc:1074] tablet fac8d3842d994279b355f9a18a82f901: '745ac12076e84effa5bde1ecbd6a32ec' -> '6f81f803ec7746ccb502c59b7cbf8eb2' move scheduled
I20240501 14:01:18.887970 8724 rebalancer_tool.cc:1136] tablet 205af5386f904e70989e647da22750e8: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move completed: OK
I20240501 14:01:18.897840 8724 rebalancer_tool.cc:1136] tablet 6904a974640a4e84a80b1fb0749365b7: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:19.038597 8724 rebalancer_tool.cc:1074] tablet 2efec972a06b48b99eb89d454fa214a3: 'a64f34785a7b44fa8070b0452a4e0ddd' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move scheduled
I20240501 14:01:19.126688 8724 rebalancer_tool.cc:1074] tablet f8ae4d9e24a94b878627722ad94c0e35: 'a64f34785a7b44fa8070b0452a4e0ddd' -> 'f8c60f2bbac541dcba4424ab726f9ad9' move scheduled
I20240501 14:01:19.225235 8724 rebalancer_tool.cc:1136] tablet 0d4edc28f28641d4a3201cd896d97afe: 'f8c60f2bbac541dcba4424ab726f9ad9' -> 'b3da1f36f37e4a1aa9f280c2fd0e1fba' move completed: OK
I20240501 14:01:19.912849 8724 rebalancer_tool.cc:1123] tablet 8e605c78442442dd82c598bac2fd84cd: 'c9f94ddc5ff64ad79b9f01846a20e2fd' -> '745ac12076e84effa5bde1ecbd6a32ec' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:19.966583 8724 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): unacceptable health status UNAVAILABLE
I20240501 14:01:20.827242 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2065
I20240501 14:01:20.828732 11106 raft_consensus.cc:482] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.828893 11106 raft_consensus.cc:504] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:20.829542 11106 leader_election.cc:288] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
I20240501 14:01:20.830087 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
W20240501 14:01:20.830749 5229 leader_election.cc:334] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.830842 5229 leader_election.cc:302] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: c9f94ddc5ff64ad79b9f01846a20e2fd, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.831051 11106 raft_consensus.cc:2726] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:20.836169 2186 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:01:20.838302 2473 consensus_peers.cc:485] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.839874 10489 raft_consensus.cc:482] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.840548 10489 raft_consensus.cc:504] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:20.841792 10489 leader_election.cc:288] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:20.842940 5098 leader_election.cc:334] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:20.845561 2343 consensus_peers.cc:485] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.882032 10234 raft_consensus.cc:482] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.882205 10234 raft_consensus.cc:504] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:20.882882 10234 leader_election.cc:288] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:20.885193 2472 leader_election.cc:334] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:20.885571 2473 leader_election.cc:334] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:20.885663 2473 leader_election.cc:302] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.885872 10234 raft_consensus.cc:2726] T c685294d3fca4677bc1b1cf355dac3ce P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:20.891798 10700 raft_consensus.cc:482] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.891952 10700 raft_consensus.cc:504] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:20.892544 10700 leader_election.cc:288] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:20.894049 2473 leader_election.cc:334] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:20.894184 2472 leader_election.cc:334] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.894325 2472 leader_election.cc:302] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.894443 11221 raft_consensus.cc:482] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.894636 10700 raft_consensus.cc:2726] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:20.894647 11221 raft_consensus.cc:504] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:20.895917 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697989873e94a6fad3835e0ebb36fce" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:20.896117 5293 raft_consensus.cc:2454] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:01:20.896251 11221 leader_election.cc:288] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:20.896564 2343 leader_election.cc:302] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
I20240501 14:01:20.896795 11221 raft_consensus.cc:2781] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:20.896852 11221 raft_consensus.cc:482] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.896943 11221 raft_consensus.cc:3037] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:20.898880 2342 leader_election.cc:334] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.898932 11221 raft_consensus.cc:504] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:20.899864 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3697989873e94a6fad3835e0ebb36fce" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:20.900050 5293 raft_consensus.cc:3037] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:20.900300 11221 leader_election.cc:288] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Requested vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:20.903745 2342 leader_election.cc:334] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.904495 5293 raft_consensus.cc:2454] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 2.
W20240501 14:01:20.904516 5230 consensus_peers.cc:485] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:20.904860 2343 leader_election.cc:302] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, a64f34785a7b44fa8070b0452a4e0ddd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:20.905484 11221 raft_consensus.cc:2781] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:20.906867 11217 raft_consensus.cc:482] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.907080 11221 raft_consensus.cc:686] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [term 2 LEADER]: Becoming Leader. State: Replica: a64f34785a7b44fa8070b0452a4e0ddd, State: Running, Role: LEADER
I20240501 14:01:20.907012 11217 raft_consensus.cc:504] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:20.907315 11221 consensus_queue.cc:237] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:20.907635 11217 leader_election.cc:288] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:20.909144 2212 leader_election.cc:334] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:20.919764 2006 catalog_manager.cc:5484] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd reported cstate change: term changed from 1 to 2, leader changed from f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) to a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3). New cstate: current_term: 2 leader_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:20.970944 11221 raft_consensus.cc:482] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.971271 11221 raft_consensus.cc:504] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:20.972432 11221 leader_election.cc:288] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:20.973044 2342 leader_election.cc:334] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:20.979414 2343 consensus_peers.cc:485] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:20.993476 2473 consensus_peers.cc:485] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:20.993605 2473 consensus_peers.cc:485] T 4aa75617adf64020af85f4cf2bd13917 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:20.996098 10700 raft_consensus.cc:482] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:20.996246 10700 raft_consensus.cc:504] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { replace: true } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:20.997387 10700 leader_election.cc:288] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:20.997474 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e605c78442442dd82c598bac2fd84cd" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:20.997669 5162 raft_consensus.cc:2454] T 8e605c78442442dd82c598bac2fd84cd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 2.
W20240501 14:01:20.999181 5230 consensus_peers.cc:485] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:20.999595 2472 leader_election.cc:334] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:20.999725 2473 leader_election.cc:334] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:20.999807 2473 leader_election.cc:302] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.000159 10700 raft_consensus.cc:2726] T 8e605c78442442dd82c598bac2fd84cd P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:21.004797 10489 raft_consensus.cc:482] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.004989 10489 raft_consensus.cc:504] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:21.005717 10489 leader_election.cc:288] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
W20240501 14:01:21.006915 5098 leader_election.cc:334] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.013713 11221 raft_consensus.cc:482] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.013887 11221 raft_consensus.cc:504] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:21.014744 11221 leader_election.cc:288] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:21.016345 2342 leader_election.cc:334] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.016811 10700 raft_consensus.cc:482] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.016960 10700 raft_consensus.cc:504] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:21.017504 10700 leader_election.cc:288] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:21.017489 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:21.017680 5293 raft_consensus.cc:2454] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 1.
I20240501 14:01:21.018072 5099 leader_election.cc:302] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.018348 11216 raft_consensus.cc:2781] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:21.018447 11216 raft_consensus.cc:482] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.018555 11216 raft_consensus.cc:3037] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:21.019541 5099 consensus_peers.cc:485] T fb57ae3a353f4f9e94a47a72ad8792b4 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:21.020164 2472 leader_election.cc:334] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.021533 11216 raft_consensus.cc:504] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:21.022147 11216 leader_election.cc:288] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 election: Requested vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221)
I20240501 14:01:21.022789 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c565cfcf616445ff949dd75f8f513e88" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c"
I20240501 14:01:21.022972 5293 raft_consensus.cc:3037] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:21.024492 5098 leader_election.cc:334] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:21.024766 2213 consensus_peers.cc:485] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:21.026300 5293 raft_consensus.cc:2454] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f94ddc5ff64ad79b9f01846a20e2fd in term 2.
I20240501 14:01:21.026575 10782 raft_consensus.cc:482] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.026751 10782 raft_consensus.cc:504] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:21.027367 10782 leader_election.cc:288] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:21.027354 5099 leader_election.cc:302] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.027623 11216 raft_consensus.cc:2781] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:21.027732 11216 raft_consensus.cc:686] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 LEADER]: Becoming Leader. State: Replica: c9f94ddc5ff64ad79b9f01846a20e2fd, State: Running, Role: LEADER
I20240501 14:01:21.027945 11216 consensus_queue.cc:237] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
W20240501 14:01:21.029584 2342 leader_election.cc:334] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.029760 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527b1cd4c73749f39b8659312df939ff" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:21.030315 2341 leader_election.cc:302] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 745ac12076e84effa5bde1ecbd6a32ec, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.030606 11221 raft_consensus.cc:2726] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:21.034201 5099 consensus_peers.cc:485] T 36f03024767442ac911a8a44f7ecdceb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:01:21.035295 2006 catalog_manager.cc:5484] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd reported cstate change: term changed from 1 to 2, leader changed from f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) to c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7). New cstate: current_term: 2 leader_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:21.036801 2213 consensus_peers.cc:485] T d42a56dd51fe410cb42dd284ecbf2748 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:21.037962 10700 raft_consensus.cc:482] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.038102 10700 raft_consensus.cc:504] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:21.038690 10700 leader_election.cc:288] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:21.038812 11216 raft_consensus.cc:482] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.038940 11216 raft_consensus.cc:504] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:21.039237 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527b1cd4c73749f39b8659312df939ff" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:21.039422 2406 raft_consensus.cc:2454] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 1.
I20240501 14:01:21.039547 11216 leader_election.cc:288] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:21.039780 2471 leader_election.cc:302] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters:
W20240501 14:01:21.040738 2472 leader_election.cc:334] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.041072 10700 raft_consensus.cc:2781] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:21.041137 10700 raft_consensus.cc:482] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.041186 10700 raft_consensus.cc:3037] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:21.041117 5098 leader_election.cc:334] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.043380 10700 raft_consensus.cc:504] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:21.044212 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527b1cd4c73749f39b8659312df939ff" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd"
I20240501 14:01:21.044397 2406 raft_consensus.cc:3037] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:21.045326 10700 leader_election.cc:288] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 election: Requested vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:21.045935 2472 leader_election.cc:334] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.046145 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
I20240501 14:01:21.046638 5099 leader_election.cc:302] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 5a4b9a96e9734baa8a03d6056bfa934c, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.046900 11216 raft_consensus.cc:2726] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:21.046742 2406 raft_consensus.cc:2454] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745ac12076e84effa5bde1ecbd6a32ec in term 2.
I20240501 14:01:21.047469 2471 leader_election.cc:302] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.047709 10700 raft_consensus.cc:2781] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:21.048610 10700 raft_consensus.cc:686] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [term 2 LEADER]: Becoming Leader. State: Replica: 745ac12076e84effa5bde1ecbd6a32ec, State: Running, Role: LEADER
I20240501 14:01:21.048919 10700 consensus_queue.cc:237] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } }
I20240501 14:01:21.053504 11221 raft_consensus.cc:482] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.053669 11221 raft_consensus.cc:504] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:21.053817 2006 catalog_manager.cc:5484] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec reported cstate change: term changed from 1 to 2, leader changed from f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) to 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4). New cstate: current_term: 2 leader_uuid: "745ac12076e84effa5bde1ecbd6a32ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:21.054179 11221 leader_election.cc:288] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:21.055331 2343 leader_election.cc:334] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:21.056320 2342 leader_election.cc:334] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.056427 2342 leader_election.cc:302] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.056650 11221 raft_consensus.cc:2726] T ea342bfcde3b4daa8dcc39126fd42cdc P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:01:21.067170 2472 consensus_peers.cc:462] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.079293 11106 raft_consensus.cc:482] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.079466 11106 raft_consensus.cc:504] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:21.080075 11106 leader_election.cc:288] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:21.080494 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
I20240501 14:01:21.080696 5162 raft_consensus.cc:2454] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:01:21.081382 5228 leader_election.cc:302] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters:
W20240501 14:01:21.081524 5229 leader_election.cc:334] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.081729 11106 raft_consensus.cc:2781] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:21.081811 11106 raft_consensus.cc:482] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.081885 11106 raft_consensus.cc:3037] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:21.083772 11106 raft_consensus.cc:504] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:21.084324 11106 leader_election.cc:288] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: Requested vote from peers c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:21.084656 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8502afe4b8a7439b882961145c9475f5" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd"
I20240501 14:01:21.084834 5162 raft_consensus.cc:3037] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:01:21.085587 5229 leader_election.cc:334] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.086820 5162 raft_consensus.cc:2454] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 2.
I20240501 14:01:21.087190 5228 leader_election.cc:302] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.087406 11106 raft_consensus.cc:2781] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:21.087527 11106 raft_consensus.cc:686] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 LEADER]: Becoming Leader. State: Replica: 5a4b9a96e9734baa8a03d6056bfa934c, State: Running, Role: LEADER
I20240501 14:01:21.087735 11106 consensus_queue.cc:237] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:21.093255 2006 catalog_manager.cc:5484] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c reported cstate change: term changed from 1 to 2, leader changed from f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5) to 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8). New cstate: current_term: 2 leader_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:21.095808 2213 consensus_peers.cc:485] T 14e43f9e141140be88a7dc1fa8963c49 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:21.106755 10234 raft_consensus.cc:482] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.106916 10234 raft_consensus.cc:504] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:21.107532 10234 leader_election.cc:288] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:21.109025 2472 leader_election.cc:334] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:21.109169 2473 leader_election.cc:334] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:21.109275 2473 leader_election.cc:302] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.109470 10234 raft_consensus.cc:2726] T 86abd8c2e86044f59c8074aa9a77db0d P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:21.113840 2213 consensus_peers.cc:485] T 782c96bf253c4b09bc9c693d2a553b00 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:21.172238 2343 consensus_peers.cc:485] T f1e048df08cf4f7eaacb78d6592d79b1 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:01:21.189181 10946 raft_consensus.cc:482] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
W20240501 14:01:21.189867 1992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240501 14:01:21.191062 10946 raft_consensus.cc:504] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:21.191686 10946 leader_election.cc:288] T aaaae937ea4d44068d3ad5b9024b9421 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
I20240501 14:01:21.191799 11175 raft_consensus.cc:482] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:21.191897 11175 raft_consensus.cc:504] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
W20240501 14:01:21.192242 1992 catalog_manager.cc:4573] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b9ccb0f16d8044da9f3f353ffbea5ce9: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:21.192361 11175 leader_election.cc:288] T 9a74335efeb04521baf1720a14093643 P d1a414d381f44829a2510cb76d481887 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:21.194347 2146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ccb0f16d8044da9f3f353ffbea5ce9" candidate_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1a414d381f44829a2510cb76d481887" is_pre_election: true
I20240501 14:01:21.194550 2146 raft_consensus.cc:2454] T b9ccb0f16d8044da9f3f353ffbea5ce9 P d1a414d381f44829a2510cb76d481887 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a64f34785a7b44fa8070b0452a4e0ddd in term 1.
I20240501 14:01:21.194640 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2065
I20240501 14:01:21.195521 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a74335efeb04521baf1720a14093643" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:21.195739 2537 raft_consensus.cc:2454] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:21.197913 2405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaaae937ea4d44068d3ad5b9024b9421" candidate_uuid: "d1a414d381f44829a2510cb76d481887" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" is_pre_election: true
I20240501 14:01:21.198159 2405 raft_consensus.cc:2454] T aaaae937ea4d44068d3ad5b9024b9421 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a414d381f44829a2510cb76d481887 in term 1.
I20240501 14:01:21.207650 5162 raft_consensus.cc:1261] T 9eec5cc39eda4f6d85f83959952a322c P c9f94ddc5ff64ad79b9f01846a20e2fd [term 3 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240501 14:01:21.208345 10234 consensus_queue.cc:1035] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:01:21.217332 2472 consensus_peers.cc:485] T 9eec5cc39eda4f6d85f83959952a322c P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.245280 2341 connection.cc:535] server connection from 127.28.17.1:37413 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.245343 5098 connection.cc:535] server connection from 127.28.17.1:40345 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.245473 2213 connection.cc:535] server connection from 127.28.17.1:47297 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.245568 2471 connection.cc:535] server connection from 127.28.17.1:53971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.245745 2471 connection.cc:535] client connection to 127.28.17.1:41309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.245846 2211 connection.cc:535] client connection to 127.28.17.1:41309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.246539 5097 connection.cc:535] client connection to 127.28.17.1:41309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:21.246764 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2196
W20240501 14:01:21.247125 2211 leader_election.cc:334] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:21.247227 2211 leader_election.cc:302] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3da1f36f37e4a1aa9f280c2fd0e1fba; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
W20240501 14:01:21.247509 2211 consensus_peers.cc:485] T 238aa15a2cd74ff69e17a2934b32b70d P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.247583 2211 consensus_peers.cc:485] T 8e52258f0b8f4bcdaebc75618bd2baa3 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.247633 2211 consensus_peers.cc:485] T 6fe1b54933ad446082b1d2840b2c6e49 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.247681 2211 consensus_peers.cc:485] T 85842504df3e4571b1a9750809711db2 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.247757 2211 consensus_peers.cc:485] T 566acdef1d114b8aac9b55ccfa03da35 P b3da1f36f37e4a1aa9f280c2fd0e1fba -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.248122 2341 connection.cc:535] client connection to 127.28.17.1:41309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:21.248370 10640 raft_consensus.cc:2726] T 2e20cd9040e4448b93ac513371a7fcf9 P b3da1f36f37e4a1aa9f280c2fd0e1fba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:21.249898 5097 consensus_peers.cc:485] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.250027 5097 consensus_peers.cc:485] T 13d8ec926b7a46568470e824763569e6 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.250106 5097 leader_election.cc:334] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
W20240501 14:01:21.250154 2341 leader_election.cc:334] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:21.250198 5097 leader_election.cc:302] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.250276 2341 leader_election.cc:302] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
W20240501 14:01:21.250587 2341 leader_election.cc:334] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:21.250675 2341 leader_election.cc:302] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a64f34785a7b44fa8070b0452a4e0ddd; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.250792 11216 raft_consensus.cc:2726] T afc49574262e4819b6c86025c43c772a P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:21.250909 2341 consensus_peers.cc:485] T 46abd9f84a1c45f4b54447990d8ba65b P a64f34785a7b44fa8070b0452a4e0ddd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.250989 2341 consensus_peers.cc:485] T d5086890bd254edebdf12812ff5c3728 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.251041 2341 consensus_peers.cc:485] T 457c62e1ddf547ad900ed529b3244474 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.251108 2341 consensus_peers.cc:485] T 741357494d3a4842935671afd1dbe677 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:21.251298 10782 raft_consensus.cc:2726] T b9ccb0f16d8044da9f3f353ffbea5ce9 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:21.251461 10782 raft_consensus.cc:2726] T 1d2f1ffd51584255b020ed742f30f573 P a64f34785a7b44fa8070b0452a4e0ddd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:21.252151 2471 consensus_peers.cc:485] T eb1fa60c64034b7d8c7b4879823b0d83 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.252272 2471 consensus_peers.cc:485] T 1298bb5c8df3430b95d01ad23ff9cb71 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.252360 2471 consensus_peers.cc:485] T 6d8dfee2d66f44ba9ebcf74f6e570230 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.252422 2471 consensus_peers.cc:485] T 87a79bfca7cd4ffcbb0e4448320d0576 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.252482 2471 consensus_peers.cc:485] T b9ce2e195d834f6b986ac7273e7c6b78 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.252545 2471 leader_election.cc:334] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:21.252624 2471 leader_election.cc:302] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.252853 10234 raft_consensus.cc:2726] T 9a74335efeb04521baf1720a14093643 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:21.255853 2471 consensus_peers.cc:485] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:21.256734 2317 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:01:21.270977 5230 consensus_peers.cc:485] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:21.272503 2343 consensus_peers.cc:485] T fac8d3842d994279b355f9a18a82f901 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:21.302649 2341 consensus_peers.cc:485] T 5b1f151470f24501857f38461d5b70c8 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.324100 2471 consensus_peers.cc:485] T f6e71ec5dcbe4a08a0a7da5944dfe77d P 745ac12076e84effa5bde1ecbd6a32ec -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:21.460649 5293 raft_consensus.cc:1261] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Refusing update from remote peer a64f34785a7b44fa8070b0452a4e0ddd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:21.461215 10782 consensus_queue.cc:1035] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:21.477631 2342 consensus_peers.cc:485] T 3697989873e94a6fad3835e0ebb36fce P a64f34785a7b44fa8070b0452a4e0ddd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.506474 2473 consensus_peers.cc:485] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:01:21.508291 2341 consensus_peers.cc:485] T f67ca7eac95d4a7fbd28667bdde44a22 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Couldn't send request to peer d1a414d381f44829a2510cb76d481887. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.523782 2472 consensus_peers.cc:485] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:21.596573 5293 raft_consensus.cc:1261] T c565cfcf616445ff949dd75f8f513e88 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Refusing update from remote peer c9f94ddc5ff64ad79b9f01846a20e2fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:21.597167 11216 consensus_queue.cc:1035] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:21.602855 5098 consensus_peers.cc:485] T c565cfcf616445ff949dd75f8f513e88 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.617699 5229 consensus_peers.cc:485] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:21.638348 2405 raft_consensus.cc:1261] T 527b1cd4c73749f39b8659312df939ff P a64f34785a7b44fa8070b0452a4e0ddd [term 2 FOLLOWER]: Refusing update from remote peer 745ac12076e84effa5bde1ecbd6a32ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:21.639406 10228 consensus_queue.cc:1035] T 527b1cd4c73749f39b8659312df939ff P 745ac12076e84effa5bde1ecbd6a32ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:21.645565 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2196
W20240501 14:01:21.651386 2472 consensus_peers.cc:462] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.684628 5164 raft_consensus.cc:1261] T 8502afe4b8a7439b882961145c9475f5 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Refusing update from remote peer 5a4b9a96e9734baa8a03d6056bfa934c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:21.685240 11106 consensus_queue.cc:1035] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:01:21.704785 5097 connection.cc:535] client connection to 127.28.17.2:46301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.704787 5228 connection.cc:535] client connection to 127.28.17.2:46301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.705921 2341 connection.cc:535] client connection to 127.28.17.2:46301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:21.706068 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:21.705519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.17.2:46301 (local address 127.28.17.8:32997)
0501 14:01:21.705823 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:21.705900 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":199}
W20240501 14:01:21.706231 5228 consensus_peers.cc:485] T b911b0ff6ef446d8a6637edca57e570d P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.706331 5228 consensus_peers.cc:485] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.706403 5228 consensus_peers.cc:485] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:21.706418 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2326
W20240501 14:01:21.706471 5228 consensus_peers.cc:485] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.707616 2341 consensus_peers.cc:485] T bcf596d2da8849ed9b236efea09f32d9 P a64f34785a7b44fa8070b0452a4e0ddd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.707710 5097 consensus_peers.cc:485] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.707818 5097 consensus_peers.cc:485] T cdb14f12e45440b6ba991534c9628eba P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.707950 5097 consensus_peers.cc:485] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.708014 5097 consensus_peers.cc:485] T 44a4e92b144f47b5800506f4bc3eadb6 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.708076 5097 consensus_peers.cc:485] T fdb2924e91fb4a98aaa6bf977f682fdb P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.708132 5097 consensus_peers.cc:485] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.708182 5097 consensus_peers.cc:485] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.708231 5097 consensus_peers.cc:485] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:21.717007 2448 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:01:21.777402 5230 consensus_peers.cc:485] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Couldn't send request to peer 6f81f803ec7746ccb502c59b7cbf8eb2. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:01:21.778748 5228 consensus_peers.cc:485] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.801793 2471 consensus_peers.cc:485] T 2ee13db8a347476fbdabdb402c93b584 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.840585 2471 consensus_peers.cc:485] T e081778a0f67486a904f3bed7dfe8644 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.846390 2471 consensus_peers.cc:485] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.882107 2471 consensus_peers.cc:485] T ff4b3345715c44c091ed1073a8a706ff P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.894734 2471 consensus_peers.cc:485] T 98e027e2a36541eda1a67c0a3ab48d65 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:21.902843 2472 consensus_peers.cc:485] T 304b465c0a99456790bfc8f262b61ecd P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:21.908191 11258 raft_consensus.cc:482] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:21.908401 11258 raft_consensus.cc:504] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:21.909063 11258 leader_election.cc:288] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:21.910573 5228 leader_election.cc:334] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
W20240501 14:01:21.910717 5229 leader_election.cc:334] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:21.910821 5229 leader_election.cc:302] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.911026 11258 raft_consensus.cc:2726] T 27abced2b6b746d2b7c45883a328a3d3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:21.933684 11216 raft_consensus.cc:482] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:21.933861 11216 raft_consensus.cc:504] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:21.934487 11216 leader_election.cc:288] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:21.936141 5099 leader_election.cc:334] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:21.936381 5097 leader_election.cc:334] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:21.936504 5097 leader_election.cc:302] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887
I20240501 14:01:21.936748 11216 raft_consensus.cc:2726] T 913ae5fda9e842e1b3a6d7dc418a5bfd P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:21.949241 11216 raft_consensus.cc:482] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:21.949432 11216 raft_consensus.cc:504] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:21.950254 11216 leader_election.cc:288] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:21.951875 5098 leader_election.cc:334] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
W20240501 14:01:21.952302 5097 leader_election.cc:334] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:21.952414 5097 leader_election.cc:302] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: d1a414d381f44829a2510cb76d481887, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:21.952643 11216 raft_consensus.cc:2726] T fd3f8661a13944098a18d3eea7ddeb00 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:21.963243 10228 raft_consensus.cc:482] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:21.963438 10228 raft_consensus.cc:504] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:21.964248 10228 leader_election.cc:288] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:21.965328 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
W20240501 14:01:21.965927 2471 leader_election.cc:334] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
W20240501 14:01:21.966087 2473 leader_election.cc:334] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:21.966204 2473 leader_election.cc:302] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 745ac12076e84effa5bde1ecbd6a32ec; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887
I20240501 14:01:21.966420 10228 raft_consensus.cc:2726] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:01:21.999089 2472 consensus_peers.cc:485] T ead586b9a8024a218c55d2bb579a16de P 745ac12076e84effa5bde1ecbd6a32ec -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:22.054476 2471 consensus_peers.cc:485] T 9b91e5a14d82419cb295439f577d2786 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.061949 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2326
W20240501 14:01:22.064592 2471 consensus_peers.cc:485] T e4a0a92e0d48493fac254f1b249a63cf P 745ac12076e84effa5bde1ecbd6a32ec -> Peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Couldn't send request to peer b3da1f36f37e4a1aa9f280c2fd0e1fba. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.093582 11216 raft_consensus.cc:482] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.093750 11216 raft_consensus.cc:504] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } }
I20240501 14:01:22.094323 11216 leader_election.cc:288] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:22.095752 5097 leader_election.cc:334] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.095943 5097 leader_election.cc:334] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:22.096022 5097 leader_election.cc:302] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.096248 11216 raft_consensus.cc:2726] T 3efe831b65694e75b466ec2abb082590 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:22.107534 5228 connection.cc:535] client connection to 127.28.17.3:33537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:22.108013 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 2456
W20240501 14:01:22.108469 5097 connection.cc:535] client connection to 127.28.17.3:33537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:22.109297 5228 consensus_peers.cc:485] T c88cc21c0e0743999ee493f5ee856aef P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.109426 5228 consensus_peers.cc:485] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.109494 5228 consensus_peers.cc:485] T 0d4edc28f28641d4a3201cd896d97afe P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.109568 5228 consensus_peers.cc:485] T 3bc3084368de4add8809f10451d96894 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.109629 5228 consensus_peers.cc:485] T 6904a974640a4e84a80b1fb0749365b7 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.109681 5228 consensus_peers.cc:485] T 0724625a6a8e4ce38e8087472b856c5c P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.109937 2471 consensus_peers.cc:485] T f8ae4d9e24a94b878627722ad94c0e35 P 745ac12076e84effa5bde1ecbd6a32ec -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.110065 2471 consensus_peers.cc:485] T 14dc7387f47942cfa5e96aeac0805a5f P 745ac12076e84effa5bde1ecbd6a32ec -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.112082 5097 consensus_peers.cc:485] T 4ed332b3f7304bf69bc348d230ece128 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.112192 5097 consensus_peers.cc:485] T efbd148f37bb46f0a4bf0fc562fff597 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.112270 5097 consensus_peers.cc:485] T 34b137c6ac1947cd80aee3c6a6f9d138 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.112330 5097 consensus_peers.cc:485] T bc808ec4077a42d186aba7d4af5ac158 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.116613 2577 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:22.162400 10825 raft_consensus.cc:482] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.162598 10825 raft_consensus.cc:504] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:22.163311 10825 leader_election.cc:288] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:22.164839 5228 leader_election.cc:334] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
W20240501 14:01:22.169306 5098 consensus_peers.cc:485] T 0ff679e55a5446099efbdbcd1109f245 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:22.200785 11106 raft_consensus.cc:482] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.200968 11106 raft_consensus.cc:504] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } attrs { promote: false } }
I20240501 14:01:22.201651 11106 leader_election.cc:288] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:22.202728 5230 leader_election.cc:334] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:22.203485 5228 leader_election.cc:334] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:22.203598 5228 leader_election.cc:302] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.203799 11106 raft_consensus.cc:2726] T ce213e03db7f4f17913aad989c584607 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:01:22.223104 5229 consensus_peers.cc:485] T 3dff7dc57d1e4b0493778a54bc2c2c5b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:22.225524 11106 raft_consensus.cc:482] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.225668 11106 raft_consensus.cc:504] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } }
I20240501 14:01:22.226197 11106 leader_election.cc:288] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
W20240501 14:01:22.227104 5097 consensus_peers.cc:485] T 95d15ba032f64cfb8422446df83094b8 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Couldn't send request to peer a64f34785a7b44fa8070b0452a4e0ddd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.227753 5228 leader_election.cc:334] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
W20240501 14:01:22.227877 5230 leader_election.cc:334] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:22.227982 5230 leader_election.cc:302] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.228178 11106 raft_consensus.cc:2726] T a56d8b76fc244a86b73a741a937bd1c6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:22.304333 11216 raft_consensus.cc:482] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.304500 11216 raft_consensus.cc:504] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: NON_VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: true } }
I20240501 14:01:22.305078 11216 leader_election.cc:288] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
W20240501 14:01:22.306529 5097 leader_election.cc:334] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
W20240501 14:01:22.306716 5097 leader_election.cc:334] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111)
I20240501 14:01:22.306792 5097 leader_election.cc:302] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: a64f34785a7b44fa8070b0452a4e0ddd, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.307011 11216 raft_consensus.cc:2726] T 2efec972a06b48b99eb89d454fa214a3 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:22.333356 10825 raft_consensus.cc:482] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.333571 10825 raft_consensus.cc:504] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:22.334275 10825 leader_election.cc:288] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:22.335712 5228 leader_election.cc:334] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.335884 5228 leader_election.cc:334] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:22.335978 5228 leader_election.cc:302] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.336400 10825 raft_consensus.cc:2726] T b6c38a8bcb4a4e07aa424fd074d5e17e P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:22.348019 5098 consensus_peers.cc:485] T 0689911a6a8147b983ca14edee551aa0 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:22.358963 11106 raft_consensus.cc:482] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.359134 11106 raft_consensus.cc:504] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:22.359694 11106 leader_election.cc:288] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:22.361151 5228 leader_election.cc:334] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.361333 5228 leader_election.cc:334] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:22.361413 5228 leader_election.cc:302] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.361618 11106 raft_consensus.cc:2726] T 566acdef1d114b8aac9b55ccfa03da35 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:22.362390 11216 raft_consensus.cc:482] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:22.362573 11216 raft_consensus.cc:504] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:22.363265 11216 leader_election.cc:288] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:22.364451 5099 leader_election.cc:334] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:22.364571 5098 leader_election.cc:334] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:22.364704 5098 leader_election.cc:302] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:22.364909 11216 raft_consensus.cc:2726] T eb9790fa13fe4cb286f0fb7f730da336 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:01:22.373602 11106 raft_consensus.cc:482] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.373801 11106 raft_consensus.cc:504] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:22.374536 11106 leader_election.cc:288] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
W20240501 14:01:22.375954 5230 leader_election.cc:334] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:22.376209 5228 leader_election.cc:334] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
I20240501 14:01:22.376293 5228 leader_election.cc:302] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:22.376513 11106 raft_consensus.cc:2726] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 14:01:22.376531 5228 leader_election.cc:334] T 67da9784350e49209e979e232a79cfc1 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111)
I20240501 14:01:22.400477 11106 raft_consensus.cc:482] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.400676 11106 raft_consensus.cc:504] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:22.401285 11106 leader_election.cc:288] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
W20240501 14:01:22.402652 5230 leader_election.cc:334] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
W20240501 14:01:22.403395 5228 leader_election.cc:334] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
I20240501 14:01:22.403491 5228 leader_election.cc:302] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:22.403710 11106 raft_consensus.cc:2726] T 7655e45073d947eabec36e6de7d6a688 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:22.416760 11216 raft_consensus.cc:482] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.416972 11216 raft_consensus.cc:504] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:22.417702 11216 leader_election.cc:288] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:22.419219 5097 leader_election.cc:334] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.419405 5097 leader_election.cc:334] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:22.419481 5097 leader_election.cc:302] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9f94ddc5ff64ad79b9f01846a20e2fd; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.419714 11216 raft_consensus.cc:2726] T 238aa15a2cd74ff69e17a2934b32b70d P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:22.423168 11106 raft_consensus.cc:482] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.423343 11106 raft_consensus.cc:504] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:22.423915 11106 leader_election.cc:288] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
W20240501 14:01:22.425643 5228 leader_election.cc:334] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.425838 5228 leader_election.cc:334] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:22.425931 5228 leader_election.cc:302] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.426172 11106 raft_consensus.cc:2726] T 85842504df3e4571b1a9750809711db2 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:22.451807 5229 consensus_peers.cc:485] T d841d51e617d44c8adf08658c69d957b P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:01:22.452900 11274 raft_consensus.cc:482] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.453117 11274 raft_consensus.cc:504] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } }
I20240501 14:01:22.453838 11274 leader_election.cc:288] T 20cd7aaca5434c2dbb63e7111feed3a9 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:22.455835 11274 raft_consensus.cc:482] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.455971 11274 raft_consensus.cc:504] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { replace: true } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: NON_VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: true } }
I20240501 14:01:22.456622 11274 leader_election.cc:288] T e3fe9fe1c13743478e09f5eeeb528b57 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:22.456725 11274 raft_consensus.cc:482] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:22.456813 11274 raft_consensus.cc:504] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:22.457332 11274 leader_election.cc:288] T 83392c08bf314fcd98833335c247ccb1 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:22.459239 10700 raft_consensus.cc:482] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.459389 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a9e4400b4a4a4694e7449e902828c0" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:22.459403 10700 raft_consensus.cc:504] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:22.459905 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce213e03db7f4f17913aad989c584607" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "745ac12076e84effa5bde1ecbd6a32ec" is_pre_election: true
I20240501 14:01:22.460093 2536 raft_consensus.cc:2454] T ce213e03db7f4f17913aad989c584607 P 745ac12076e84effa5bde1ecbd6a32ec [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 2.
I20240501 14:01:22.460187 10700 leader_election.cc:288] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:22.460677 2537 raft_consensus.cc:2454] T 11a9e4400b4a4a4694e7449e902828c0 P 745ac12076e84effa5bde1ecbd6a32ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9a96e9734baa8a03d6056bfa934c in term 1.
I20240501 14:01:22.462968 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 2456
I20240501 14:01:22.463182 5228 leader_election.cc:302] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c, 745ac12076e84effa5bde1ecbd6a32ec; no voters: d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.463940 11258 raft_consensus.cc:2781] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:01:22.464017 11258 raft_consensus.cc:482] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.464077 11258 raft_consensus.cc:3037] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:22.465966 11258 raft_consensus.cc:504] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } }
I20240501 14:01:22.467654 5295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a9e4400b4a4a4694e7449e902828c0" candidate_uuid: "745ac12076e84effa5bde1ecbd6a32ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" is_pre_election: true
W20240501 14:01:22.467756 5228 leader_election.cc:334] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309): Network error: Client connection negotiation failed: client connection to 127.28.17.1:41309: connect: Connection refused (error 111)
I20240501 14:01:22.467878 5295 raft_consensus.cc:2379] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 745ac12076e84effa5bde1ecbd6a32ec in current term 2: Already voted for candidate 5a4b9a96e9734baa8a03d6056bfa934c in this term.
I20240501 14:01:22.466470 11258 leader_election.cc:288] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: Requested vote from peers 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117), d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309)
I20240501 14:01:22.483238 10825 raft_consensus.cc:482] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.483428 10825 raft_consensus.cc:504] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { replace: true } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:22.484282 10825 leader_election.cc:288] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
W20240501 14:01:22.485502 5230 leader_election.cc:334] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:22.486569 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30705f9c298491dbfb32bdc554647e4" candidate_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" is_pre_election: true
W20240501 14:01:22.486377 5228 leader_election.cc:334] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
I20240501 14:01:22.488473 5228 leader_election.cc:302] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 6f81f803ec7746ccb502c59b7cbf8eb2, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:22.488767 10825 raft_consensus.cc:2726] T f30705f9c298491dbfb32bdc554647e4 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:22.516361 5098 consensus_peers.cc:485] T e9a9e1d4df0a48f59b03349380cb6608 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Couldn't send request to peer f8c60f2bbac541dcba4424ab726f9ad9. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:01:22.516706 5228 connection.cc:535] client connection to 127.28.17.4:36117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:22.516815 5099 connection.cc:535] server connection from 127.28.17.4:51047 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:22.517056 5097 connection.cc:535] client connection to 127.28.17.4:36117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:01:22.517302 5228 connection.cc:535] server connection from 127.28.17.4:35953 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:22.517519 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 5082
W20240501 14:01:22.517879 5228 leader_election.cc:334] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Network error: Client connection negotiation failed: client connection to 127.28.17.4:36117: connect: Connection refused (error 111)
I20240501 14:01:22.517990 5228 leader_election.cc:302] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 745ac12076e84effa5bde1ecbd6a32ec, d1a414d381f44829a2510cb76d481887
I20240501 14:01:22.518328 10825 raft_consensus.cc:2726] T 11a9e4400b4a4a4694e7449e902828c0 P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 14:01:22.518636 5097 consensus_peers.cc:485] T 52eb73b91ed441199eade33e66b84b39 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.4:36117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.518708 5097 consensus_peers.cc:485] T 8cdc95819b944c5590202e86cc955d91 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.4:36117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.518749 5097 consensus_peers.cc:485] T ccec6fa18f844ace900db8bfe91738a5 P c9f94ddc5ff64ad79b9f01846a20e2fd -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.4:36117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.523815 5204 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:22.693605 10825 raft_consensus.cc:482] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.693800 10825 raft_consensus.cc:504] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } }
I20240501 14:01:22.694420 10825 leader_election.cc:288] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
W20240501 14:01:22.695940 5228 leader_election.cc:334] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.696100 5229 leader_election.cc:334] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:22.696218 5229 leader_election.cc:302] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, f8c60f2bbac541dcba4424ab726f9ad9
I20240501 14:01:22.696463 10825 raft_consensus.cc:2726] T 91184ecf7ec3409ca3b05318aa5dcab3 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:22.742700 10825 raft_consensus.cc:482] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.742878 10825 raft_consensus.cc:504] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } attrs { promote: false } }
I20240501 14:01:22.743479 10825 leader_election.cc:288] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537)
W20240501 14:01:22.744836 5228 leader_election.cc:334] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.745015 5228 leader_election.cc:334] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111)
I20240501 14:01:22.745090 5228 leader_election.cc:302] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: a64f34785a7b44fa8070b0452a4e0ddd, b3da1f36f37e4a1aa9f280c2fd0e1fba
I20240501 14:01:22.745301 10825 raft_consensus.cc:2726] T 878eb076eee94aa8988b3588d5395cba P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:01:22.748134 10825 raft_consensus.cc:482] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a64f34785a7b44fa8070b0452a4e0ddd)
I20240501 14:01:22.748304 10825 raft_consensus.cc:504] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:22.748925 10825 leader_election.cc:288] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:22.749197 10825 raft_consensus.cc:482] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.749312 10825 raft_consensus.cc:504] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:22.749799 10825 leader_election.cc:288] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301), c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601)
W20240501 14:01:22.750782 5228 leader_election.cc:334] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537): Network error: Client connection negotiation failed: client connection to 127.28.17.3:33537: connect: Connection refused (error 111)
W20240501 14:01:22.750952 5228 leader_election.cc:334] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Network error: Client connection negotiation failed: client connection to 127.28.17.4:36117: connect: Connection refused (error 111)
I20240501 14:01:22.751025 5228 leader_election.cc:302] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: 745ac12076e84effa5bde1ecbd6a32ec, a64f34785a7b44fa8070b0452a4e0ddd
I20240501 14:01:22.751222 10825 raft_consensus.cc:2726] T b6cca7b2b71045bf98e6eacd14e007d6 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:22.751708 5228 leader_election.cc:334] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301): Network error: Client connection negotiation failed: client connection to 127.28.17.2:46301: connect: Connection refused (error 111)
W20240501 14:01:22.774724 5228 consensus_peers.cc:485] T 1b6c6fcfebde46f18c55cf51c6b2f4f2 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.4:36117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.783865 5228 consensus_peers.cc:485] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117): Couldn't send request to peer 745ac12076e84effa5bde1ecbd6a32ec. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.4:36117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.789705 1992 catalog_manager.cc:4573] TS 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9a74335efeb04521baf1720a14093643: Network error: Client connection negotiation failed: client connection to 127.28.17.6:46037: connect: Connection refused (error 111)
I20240501 14:01:22.816082 11233 raft_consensus.cc:482] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3da1f36f37e4a1aa9f280c2fd0e1fba)
I20240501 14:01:22.816268 11233 raft_consensus.cc:504] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "b3da1f36f37e4a1aa9f280c2fd0e1fba" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 46301 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } attrs { promote: false } }
I20240501 14:01:22.817144 11233 leader_election.cc:288] T 5c4312eec255417580288818bd559099 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a4b9a96e9734baa8a03d6056bfa934c (127.28.17.8:34221), b3da1f36f37e4a1aa9f280c2fd0e1fba (127.28.17.2:46301)
I20240501 14:01:22.823247 10489 raft_consensus.cc:482] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8c60f2bbac541dcba4424ab726f9ad9)
I20240501 14:01:22.825062 11216 raft_consensus.cc:482] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1a414d381f44829a2510cb76d481887)
I20240501 14:01:22.825207 11216 raft_consensus.cc:504] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1a414d381f44829a2510cb76d481887" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 41309 } } peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "745ac12076e84effa5bde1ecbd6a32ec" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 36117 } attrs { promote: false } }
I20240501 14:01:22.823441 10489 raft_consensus.cc:504] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c9f94ddc5ff64ad79b9f01846a20e2fd" member_type: VOTER last_known_addr { host: "127.28.17.7" port: 41601 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } attrs { promote: false } }
I20240501 14:01:22.826323 11216 leader_election.cc:288] T 0fac2baeae824f68ae2fe1c62e36d970 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1a414d381f44829a2510cb76d481887 (127.28.17.1:41309), 745ac12076e84effa5bde1ecbd6a32ec (127.28.17.4:36117)
I20240501 14:01:22.826406 10489 leader_election.cc:288] T f647f3f27faf435f81e076bfc2ef0795 P c9f94ddc5ff64ad79b9f01846a20e2fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
I20240501 14:01:22.827463 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 5082
W20240501 14:01:22.868940 5228 connection.cc:535] client connection to 127.28.17.7:41601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:01:22.869356 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 5212
W20240501 14:01:22.870101 5228 consensus_peers.cc:485] T 8502afe4b8a7439b882961145c9475f5 P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.7:41601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:22.870223 5228 leader_election.cc:334] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Network error: Client connection negotiation failed: client connection to 127.28.17.7:41601: connect: Connection refused (error 111)
I20240501 14:01:22.870342 5228 leader_election.cc:302] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a4b9a96e9734baa8a03d6056bfa934c; no voters: b3da1f36f37e4a1aa9f280c2fd0e1fba, c9f94ddc5ff64ad79b9f01846a20e2fd
I20240501 14:01:22.870636 10825 raft_consensus.cc:2726] T 5c4312eec255417580288818bd559099 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 14:01:22.870703 5228 consensus_peers.cc:485] T f9b8c92907e843409e37919a1a0e96af P 5a4b9a96e9734baa8a03d6056bfa934c -> Peer c9f94ddc5ff64ad79b9f01846a20e2fd (127.28.17.7:41601): Couldn't send request to peer c9f94ddc5ff64ad79b9f01846a20e2fd. Status: Network error: Client connection negotiation failed: client connection to 127.28.17.7:41601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.875438 5335 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:23.153066 11258 raft_consensus.cc:482] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a64f34785a7b44fa8070b0452a4e0ddd)
I20240501 14:01:23.154631 11106 raft_consensus.cc:482] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a64f34785a7b44fa8070b0452a4e0ddd)
I20240501 14:01:23.154799 11106 raft_consensus.cc:504] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } }
I20240501 14:01:23.155100 11280 raft_consensus.cc:482] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a64f34785a7b44fa8070b0452a4e0ddd)
I20240501 14:01:23.155229 11280 raft_consensus.cc:504] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } attrs { promote: false } } peers { permanent_uuid: "6f81f803ec7746ccb502c59b7cbf8eb2" member_type: VOTER last_known_addr { host: "127.28.17.6" port: 46037 } attrs { promote: false } }
I20240501 14:01:23.155419 11106 leader_election.cc:288] T 6c74bacbf4464bab9c403e2153953474 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:23.155524 11258 raft_consensus.cc:504] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9a96e9734baa8a03d6056bfa934c" member_type: VOTER last_known_addr { host: "127.28.17.8" port: 34221 } } peers { permanent_uuid: "a64f34785a7b44fa8070b0452a4e0ddd" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 33537 } } peers { permanent_uuid: "f8c60f2bbac541dcba4424ab726f9ad9" member_type: VOTER last_known_addr { host: "127.28.17.5" port: 40059 } }
I20240501 14:01:23.155838 11280 leader_election.cc:288] T f1e048df08cf4f7eaacb78d6592d79b1 P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), 6f81f803ec7746ccb502c59b7cbf8eb2 (127.28.17.6:46037)
I20240501 14:01:23.156002 11258 leader_election.cc:288] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a64f34785a7b44fa8070b0452a4e0ddd (127.28.17.3:33537), f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059)
W20240501 14:01:23.157459 5229 leader_election.cc:334] T 3697989873e94a6fad3835e0ebb36fce P 5a4b9a96e9734baa8a03d6056bfa934c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8c60f2bbac541dcba4424ab726f9ad9 (127.28.17.5:40059): Network error: Client connection negotiation failed: client connection to 127.28.17.5:40059: connect: Connection refused (error 111)
I20240501 14:01:23.157698 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 5212
I20240501 14:01:23.198072 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 1975
I20240501 14:01:23.203894 2035 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:23.492674 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 1975
2024-05-01T14:01:23Z chronyd exiting
I20240501 14:01:23.552857 28740 test_util.cc:175] -----------------------------------------------
I20240501 14:01:23.552955 28740 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714571952612150-28740-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (88144 ms)
[----------] 1 test from DDLDuringRebalancingTest (88144 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240501 14:01:23.555233 28740 test_util.cc:268] Using random seed: -335844485
I20240501 14:01:23.557083 28740 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:01:23.557168 28740 ts_itest-base.cc:113] --------------
I20240501 14:01:23.557224 28740 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:01:23.557286 28740 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:01:23.557335 28740 ts_itest-base.cc:116] --------------
2024-05-01T14:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:01:23Z Disabled control of system clock
I20240501 14:01:23.587103 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:37567
--webserver_interface=127.28.17.62
--webserver_port=0
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.17.62:37567
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:01:23.916603 11304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:23.916954 11304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:23.917083 11304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:23.924561 11304 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:23.924716 11304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:01:23.924800 11304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:23.924875 11304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:01:23.924935 11304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:01:23.934048 11304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.17.62:37567
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.17.62:37567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.17.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:23.935433 11304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:23.937507 11304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:23.948179 11309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:23.948691 11310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:23.949014 11312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:23.949645 11304 server_base.cc:1034] running on GCE node
I20240501 14:01:23.950553 11304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:23.952067 11304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:23.953274 11304 hybrid_clock.cc:648] HybridClock initialized: now 1714572083953224 us; error 47 us; skew 500 ppm
I20240501 14:01:23.953622 11304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:23.956033 11304 webserver.cc:460] Webserver started at http://127.28.17.62:44863/ using document root <none> and password file <none>
I20240501 14:01:23.956772 11304 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:23.956887 11304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:23.957211 11304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:23.959915 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "06a3a1ca43b64559ad8b804c53555ce3"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-dx8c"
I20240501 14:01:23.960724 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "06a3a1ca43b64559ad8b804c53555ce3"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-dx8c"
I20240501 14:01:23.966394 11304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:01:23.969836 11318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.971216 11304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:23.971390 11304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
uuid: "06a3a1ca43b64559ad8b804c53555ce3"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-dx8c"
I20240501 14:01:23.971586 11304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:23.992700 11304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:23.993614 11304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:23.993897 11304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:24.011705 11369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.62:37567 every 8 connection(s)
I20240501 14:01:24.011699 11304 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.62:37567
I20240501 14:01:24.013211 11304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:01:24.017102 11370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.017426 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 11304
I20240501 14:01:24.017619 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:01:24.030658 11370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3: Bootstrap starting.
I20240501 14:01:24.033808 11370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.034843 11370 log.cc:824] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:24.037459 11370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3: No bootstrap required, opened a new log
I20240501 14:01:24.043742 11370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a3a1ca43b64559ad8b804c53555ce3" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 37567 } }
I20240501 14:01:24.044179 11370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.044268 11370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a3a1ca43b64559ad8b804c53555ce3, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.044821 11370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a3a1ca43b64559ad8b804c53555ce3" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 37567 } }
I20240501 14:01:24.045006 11370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:01:24.045106 11370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:01:24.045224 11370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.047338 11370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a3a1ca43b64559ad8b804c53555ce3" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 37567 } }
I20240501 14:01:24.047904 11370 leader_election.cc:302] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a3a1ca43b64559ad8b804c53555ce3; no voters:
I20240501 14:01:24.048524 11370 leader_election.cc:288] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:01:24.048625 11375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:24.049217 11375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [term 1 LEADER]: Becoming Leader. State: Replica: 06a3a1ca43b64559ad8b804c53555ce3, State: Running, Role: LEADER
I20240501 14:01:24.049727 11375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a3a1ca43b64559ad8b804c53555ce3" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 37567 } }
I20240501 14:01:24.050794 11370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:01:24.052405 11377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a3a1ca43b64559ad8b804c53555ce3. Latest consensus state: current_term: 1 leader_uuid: "06a3a1ca43b64559ad8b804c53555ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a3a1ca43b64559ad8b804c53555ce3" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 37567 } } }
I20240501 14:01:24.052733 11377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [sys.catalog]: This master's current role is: LEADER
I20240501 14:01:24.053078 11376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a3a1ca43b64559ad8b804c53555ce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a3a1ca43b64559ad8b804c53555ce3" member_type: VOTER last_known_addr { host: "127.28.17.62" port: 37567 } } }
I20240501 14:01:24.053342 11376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3 [sys.catalog]: This master's current role is: LEADER
I20240501 14:01:24.054280 11381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:01:24.058588 11381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:01:24.067584 11381 catalog_manager.cc:1347] Generated new cluster ID: aa36a1a587a84aac85664763f89a310b
I20240501 14:01:24.067716 11381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:01:24.080020 11381 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:01:24.081293 11381 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:01:24.092655 11381 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 06a3a1ca43b64559ad8b804c53555ce3: Generated new TSK 0
I20240501 14:01:24.093535 11381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:01:24.100662 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.1:0
--local_ip_for_outbound_sockets=127.28.17.1
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:01:24.430931 11394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:24.431236 11394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:24.431362 11394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:24.438935 11394 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:24.439074 11394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:24.439244 11394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.1
I20240501 14:01:24.448410 11394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.17.1
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.1
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:24.449770 11394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:24.451699 11394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:24.464546 11400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:24.464969 11399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:24.465413 11402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:24.466043 11394 server_base.cc:1034] running on GCE node
I20240501 14:01:24.466729 11394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:24.467730 11394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:24.468923 11394 hybrid_clock.cc:648] HybridClock initialized: now 1714572084468877 us; error 43 us; skew 500 ppm
I20240501 14:01:24.469259 11394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:24.471647 11394 webserver.cc:460] Webserver started at http://127.28.17.1:42279/ using document root <none> and password file <none>
I20240501 14:01:24.472425 11394 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:24.472537 11394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:24.472870 11394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:24.475666 11394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-dx8c"
I20240501 14:01:24.476497 11394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-dx8c"
I20240501 14:01:24.481802 11394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 14:01:24.485301 11408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.486744 11394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:24.486905 11394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-dx8c"
I20240501 14:01:24.487090 11394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:24.511197 11394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:24.512079 11394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:24.512351 11394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:24.513310 11394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:24.514855 11394 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:24.514946 11394 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.515058 11394 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:24.515132 11394 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.545099 11394 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.1:44355
I20240501 14:01:24.545168 11520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.1:44355 every 8 connection(s)
I20240501 14:01:24.546607 11394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:01:24.551775 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 11394
I20240501 14:01:24.551973 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:01:24.556026 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.2:0
--local_ip_for_outbound_sockets=127.28.17.2
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:24.560614 11521 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:24.560935 11521 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:24.561866 11521 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:24.564272 11335 ts_manager.cc:194] Registered new tserver with Master: 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:24.565819 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.1:49823
W20240501 14:01:24.892554 11525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:24.892856 11525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:24.892976 11525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:24.900508 11525 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:24.900642 11525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:24.900790 11525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.2
I20240501 14:01:24.909981 11525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.17.2
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.2
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:24.911370 11525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:24.913138 11525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:24.925401 11530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:24.925935 11533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:24.926301 11531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:24.926990 11525 server_base.cc:1034] running on GCE node
I20240501 14:01:24.927655 11525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:24.928709 11525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:24.929924 11525 hybrid_clock.cc:648] HybridClock initialized: now 1714572084929868 us; error 54 us; skew 500 ppm
I20240501 14:01:24.930262 11525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:24.936884 11525 webserver.cc:460] Webserver started at http://127.28.17.2:36267/ using document root <none> and password file <none>
I20240501 14:01:24.937693 11525 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:24.937816 11525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:24.938150 11525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:24.940938 11525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cf637c0b31e34127b7b2599a8c2763a6"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-dx8c"
I20240501 14:01:24.941730 11525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cf637c0b31e34127b7b2599a8c2763a6"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-dx8c"
I20240501 14:01:24.947258 11525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 14:01:24.950888 11539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.952383 11525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:24.952554 11525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cf637c0b31e34127b7b2599a8c2763a6"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-dx8c"
I20240501 14:01:24.952762 11525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:24.969380 11525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:24.970309 11525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:24.970651 11525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:24.971644 11525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:24.973222 11525 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:24.973317 11525 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.973433 11525 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:24.973500 11525 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:25.003648 11525 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.2:42357
I20240501 14:01:25.003697 11651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.2:42357 every 8 connection(s)
I20240501 14:01:25.004967 11525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:01:25.015523 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 11525
I20240501 14:01:25.015738 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:01:25.018215 11652 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:25.018553 11652 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:25.019182 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.3:0
--local_ip_for_outbound_sockets=127.28.17.3
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:25.019438 11652 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:25.020795 11335 ts_manager.cc:194] Registered new tserver with Master: cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:25.021548 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.2:55945
W20240501 14:01:25.353507 11655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:25.353811 11655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:25.353909 11655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:25.361621 11655 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:25.361752 11655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:25.361877 11655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.3
I20240501 14:01:25.371161 11655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.17.3
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.3
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:25.372491 11655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:25.374261 11655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:25.388053 11660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:25.388546 11661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:25.388875 11663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:25.389597 11655 server_base.cc:1034] running on GCE node
I20240501 14:01:25.390269 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:25.391458 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:25.392668 11655 hybrid_clock.cc:648] HybridClock initialized: now 1714572085392606 us; error 59 us; skew 500 ppm
I20240501 14:01:25.393002 11655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:25.395365 11655 webserver.cc:460] Webserver started at http://127.28.17.3:39149/ using document root <none> and password file <none>
I20240501 14:01:25.396134 11655 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:25.396245 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:25.396569 11655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:25.399281 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
format_stamp: "Formatted at 2024-05-01 14:01:25 on dist-test-slave-dx8c"
I20240501 14:01:25.400040 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
format_stamp: "Formatted at 2024-05-01 14:01:25 on dist-test-slave-dx8c"
I20240501 14:01:25.405524 11655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:01:25.409163 11669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:25.410627 11655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:01:25.410801 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
format_stamp: "Formatted at 2024-05-01 14:01:25 on dist-test-slave-dx8c"
I20240501 14:01:25.411067 11655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:25.435266 11655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:25.436132 11655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:25.436403 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:25.437345 11655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:25.438853 11655 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:25.438951 11655 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:25.439056 11655 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:25.439121 11655 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:25.468199 11655 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.3:37807
I20240501 14:01:25.468248 11781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.3:37807 every 8 connection(s)
I20240501 14:01:25.469522 11655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:01:25.472263 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 11655
I20240501 14:01:25.472445 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:01:25.475800 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.4:0
--local_ip_for_outbound_sockets=127.28.17.4
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:25.482754 11782 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:25.483052 11782 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:25.483965 11782 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:25.485487 11335 ts_manager.cc:194] Registered new tserver with Master: cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:25.486267 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.3:50079
I20240501 14:01:25.568934 11521 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
W20240501 14:01:25.805446 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:25.805770 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:25.805898 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:25.813421 11785 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:25.813582 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:25.813746 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.4
I20240501 14:01:25.822904 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.17.4
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.4
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:25.824270 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:25.825989 11785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:25.837889 11791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:25.838222 11790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:25.838815 11793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:25.839316 11785 server_base.cc:1034] running on GCE node
I20240501 14:01:25.839910 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:25.840822 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:25.842017 11785 hybrid_clock.cc:648] HybridClock initialized: now 1714572085841963 us; error 47 us; skew 500 ppm
I20240501 14:01:25.842363 11785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:25.844640 11785 webserver.cc:460] Webserver started at http://127.28.17.4:36719/ using document root <none> and password file <none>
I20240501 14:01:25.845407 11785 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:25.845510 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:25.845827 11785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:25.848582 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "662569f31808430ea81047cac005191e"
format_stamp: "Formatted at 2024-05-01 14:01:25 on dist-test-slave-dx8c"
I20240501 14:01:25.849364 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "662569f31808430ea81047cac005191e"
format_stamp: "Formatted at 2024-05-01 14:01:25 on dist-test-slave-dx8c"
I20240501 14:01:25.854868 11785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20240501 14:01:25.858471 11799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:25.859910 11785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:01:25.860093 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "662569f31808430ea81047cac005191e"
format_stamp: "Formatted at 2024-05-01 14:01:25 on dist-test-slave-dx8c"
I20240501 14:01:25.860301 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:25.883900 11785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:25.884807 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:25.885087 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:25.886075 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:25.887759 11785 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:25.887859 11785 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:25.887969 11785 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:25.888036 11785 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:25.917778 11785 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.4:33167
I20240501 14:01:25.917809 11911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.4:33167 every 8 connection(s)
I20240501 14:01:25.919158 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:01:25.928042 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 11785
I20240501 14:01:25.928244 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:01:25.931638 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:0
--local_ip_for_outbound_sockets=127.28.17.5
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:25.931864 11912 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:25.932184 11912 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:25.933012 11912 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:25.934491 11335 ts_manager.cc:194] Registered new tserver with Master: 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:25.935551 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.4:42265
I20240501 14:01:26.024111 11652 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
W20240501 14:01:26.262847 11915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:26.263139 11915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:26.263260 11915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:26.270888 11915 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:26.271040 11915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:26.271198 11915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 14:01:26.280550 11915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:26.281951 11915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:26.283782 11915 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:26.296869 11921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:26.297219 11920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:26.297746 11923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:26.298831 11915 server_base.cc:1034] running on GCE node
I20240501 14:01:26.299455 11915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:26.300443 11915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:26.301656 11915 hybrid_clock.cc:648] HybridClock initialized: now 1714572086301595 us; error 63 us; skew 500 ppm
I20240501 14:01:26.301987 11915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:26.304478 11915 webserver.cc:460] Webserver started at http://127.28.17.5:35179/ using document root <none> and password file <none>
I20240501 14:01:26.305241 11915 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:26.305357 11915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:26.305675 11915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:26.308676 11915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4de87cf6c87c429e8bd46a179d625e8c"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:26.309479 11915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4de87cf6c87c429e8bd46a179d625e8c"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:26.315034 11915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 14:01:26.318785 11929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:26.320380 11915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:26.320562 11915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4de87cf6c87c429e8bd46a179d625e8c"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:26.320816 11915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:26.350248 11915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:26.351167 11915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:26.351423 11915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:26.352478 11915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:26.354020 11915 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:26.354105 11915 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:26.354208 11915 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:26.354280 11915 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:26.393410 11915 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:33957
I20240501 14:01:26.393453 12041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:33957 every 8 connection(s)
I20240501 14:01:26.394778 11915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:01:26.403471 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 11915
I20240501 14:01:26.403657 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:01:26.407169 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.6:0
--local_ip_for_outbound_sockets=127.28.17.6
--webserver_interface=127.28.17.6
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:26.408664 12042 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:26.409019 12042 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:26.410033 12042 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:26.411844 11335 ts_manager.cc:194] Registered new tserver with Master: 4de87cf6c87c429e8bd46a179d625e8c (127.28.17.5:33957)
I20240501 14:01:26.412999 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:52959
I20240501 14:01:26.489223 11782 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
W20240501 14:01:26.757761 12045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:26.758066 12045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:26.758203 12045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:26.766098 12045 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:26.766240 12045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:26.766387 12045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.6
I20240501 14:01:26.775655 12045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.17.6
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.6
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:26.777055 12045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:26.778815 12045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:26.801656 12053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:26.802098 12050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:26.802714 12051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:26.803433 12045 server_base.cc:1034] running on GCE node
I20240501 14:01:26.804082 12045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:26.805099 12045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:26.806309 12045 hybrid_clock.cc:648] HybridClock initialized: now 1714572086806268 us; error 45 us; skew 500 ppm
I20240501 14:01:26.806707 12045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:26.808954 12045 webserver.cc:460] Webserver started at http://127.28.17.6:34589/ using document root <none> and password file <none>
I20240501 14:01:26.809710 12045 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:26.809818 12045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:26.810137 12045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:26.812773 12045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "65256cc016354123aab3c93a4f191d0f"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:26.813568 12045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "65256cc016354123aab3c93a4f191d0f"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:26.818935 12045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 14:01:26.822342 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:26.823725 12045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:26.823886 12045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "65256cc016354123aab3c93a4f191d0f"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:26.824090 12045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:26.840543 12045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:26.841415 12045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:26.841674 12045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:26.842671 12045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:26.844148 12045 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:26.844244 12045 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:26.844349 12045 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:26.844414 12045 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:26.875548 12045 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.6:42901
I20240501 14:01:26.875602 12171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.6:42901 every 8 connection(s)
I20240501 14:01:26.876906 12045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:01:26.882660 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 12045
I20240501 14:01:26.882865 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:01:26.885802 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.7:0
--local_ip_for_outbound_sockets=127.28.17.7
--webserver_interface=127.28.17.7
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:26.889410 12172 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:26.889724 12172 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:26.890623 12172 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:26.892331 11335 ts_manager.cc:194] Registered new tserver with Master: 65256cc016354123aab3c93a4f191d0f (127.28.17.6:42901)
I20240501 14:01:26.893339 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.6:38045
I20240501 14:01:26.938079 11912 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
W20240501 14:01:27.233111 12175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:27.233415 12175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:27.233547 12175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:27.241144 12175 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:27.241297 12175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:27.241463 12175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.7
I20240501 14:01:27.250777 12175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.17.7
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.7
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:27.252277 12175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:27.253995 12175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:27.273591 12181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:27.274667 12180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:27.274688 12183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:27.276249 12175 server_base.cc:1034] running on GCE node
I20240501 14:01:27.276908 12175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:27.277954 12175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:27.279218 12175 hybrid_clock.cc:648] HybridClock initialized: now 1714572087279167 us; error 36 us; skew 500 ppm
I20240501 14:01:27.279558 12175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:27.281915 12175 webserver.cc:460] Webserver started at http://127.28.17.7:45671/ using document root <none> and password file <none>
I20240501 14:01:27.282713 12175 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:27.282820 12175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:27.283154 12175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:27.285745 12175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "9e51845ef9224a8bb157cb19425907e1"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:27.286550 12175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "9e51845ef9224a8bb157cb19425907e1"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:27.292107 12175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240501 14:01:27.295655 12189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:27.297034 12175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:01:27.297200 12175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9e51845ef9224a8bb157cb19425907e1"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:27.297389 12175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:27.319626 12175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:27.320521 12175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:27.320775 12175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:27.321801 12175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:27.323458 12175 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:27.323554 12175 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:27.323659 12175 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:27.323726 12175 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:27.357437 12175 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.7:46081
I20240501 14:01:27.357474 12301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.7:46081 every 8 connection(s)
I20240501 14:01:27.358817 12175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:01:27.368984 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 12175
I20240501 14:01:27.369124 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:01:27.371436 12302 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:27.371769 12302 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:27.372551 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.8:0
--local_ip_for_outbound_sockets=127.28.17.8
--webserver_interface=127.28.17.8
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:27.372622 12302 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:27.374084 11335 ts_manager.cc:194] Registered new tserver with Master: 9e51845ef9224a8bb157cb19425907e1 (127.28.17.7:46081)
I20240501 14:01:27.374924 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.7:57683
I20240501 14:01:27.416249 12042 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
W20240501 14:01:27.708256 12305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:27.708592 12305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:27.708724 12305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:27.716323 12305 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:27.716475 12305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:27.716651 12305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.8
I20240501 14:01:27.725930 12305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.17.8
--webserver_port=0
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.8
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:27.727351 12305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:27.729244 12305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:27.741477 12311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:27.741986 12310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:27.742452 12313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:27.743206 12305 server_base.cc:1034] running on GCE node
I20240501 14:01:27.743816 12305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:27.744773 12305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:27.745977 12305 hybrid_clock.cc:648] HybridClock initialized: now 1714572087745935 us; error 42 us; skew 500 ppm
I20240501 14:01:27.746287 12305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:27.748522 12305 webserver.cc:460] Webserver started at http://127.28.17.8:44077/ using document root <none> and password file <none>
I20240501 14:01:27.749267 12305 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:27.749379 12305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:27.749692 12305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:27.752586 12305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6faa12d59f294fc39709e31aa1826aa5"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:27.753371 12305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6faa12d59f294fc39709e31aa1826aa5"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:27.759043 12305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 14:01:27.762760 12319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:27.764161 12305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:01:27.764319 12305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6faa12d59f294fc39709e31aa1826aa5"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:27.764523 12305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:27.786226 12305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:27.787180 12305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:27.787443 12305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:27.788395 12305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:27.789909 12305 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:27.790005 12305 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:27.790117 12305 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:27.790186 12305 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:27.819675 12305 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.8:36305
I20240501 14:01:27.819720 12431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.8:36305 every 8 connection(s)
I20240501 14:01:27.820950 12305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:01:27.823668 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 12305
I20240501 14:01:27.823866 28740 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:01:27.834123 12432 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:27.834432 12432 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:27.835264 12432 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:27.836633 11335 ts_manager.cc:194] Registered new tserver with Master: 6faa12d59f294fc39709e31aa1826aa5 (127.28.17.8:36305)
I20240501 14:01:27.837383 11335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.8:39063
I20240501 14:01:27.840003 28740 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:01:27.855693 28740 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:01:27.855824 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 11915
I20240501 14:01:27.863895 12037 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:27.895973 12172 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
I20240501 14:01:27.945884 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 11915
I20240501 14:01:27.959287 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 12045
I20240501 14:01:27.970567 12167 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:28.050577 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 12045
I20240501 14:01:28.063688 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 12175
I20240501 14:01:28.071816 12297 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:28.150637 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 12175
I20240501 14:01:28.166894 28740 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task84tCje/build/asan/bin/kudu pid 12305
I20240501 14:01:28.177867 12427 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:01:28.261693 28740 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task84tCje/build/asan/bin/kudu with pid 12305
I20240501 14:01:32.029703 11335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:32.068650 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 571d6b5b23424ea29dbe787584525efc (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.070322 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571d6b5b23424ea29dbe787584525efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.074090 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 35930481673f43e2bd8f29ed2ca5b317 (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.074698 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35930481673f43e2bd8f29ed2ca5b317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.075704 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 35930481673f43e2bd8f29ed2ca5b317 (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.077021 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35930481673f43e2bd8f29ed2ca5b317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.078225 11587 tablet_service.cc:1467] Processing CreateTablet for tablet 571d6b5b23424ea29dbe787584525efc (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.079113 11586 tablet_service.cc:1467] Processing CreateTablet for tablet cf124b7d872c4c2aacddf2a2ea08374d (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.079807 11587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571d6b5b23424ea29dbe787584525efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.079826 11846 tablet_service.cc:1467] Processing CreateTablet for tablet cf124b7d872c4c2aacddf2a2ea08374d (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.080358 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf124b7d872c4c2aacddf2a2ea08374d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.083942 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf124b7d872c4c2aacddf2a2ea08374d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.090445 11456 tablet_service.cc:1467] Processing CreateTablet for tablet 35930481673f43e2bd8f29ed2ca5b317 (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.089224 11454 tablet_service.cc:1467] Processing CreateTablet for tablet cf124b7d872c4c2aacddf2a2ea08374d (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.092483 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35930481673f43e2bd8f29ed2ca5b317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.093621 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf124b7d872c4c2aacddf2a2ea08374d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.094640 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 571d6b5b23424ea29dbe787584525efc (DEFAULT_TABLE table=rebalance_test_table_0 [id=686dc36c92b04ea0af2138fe2cc7b889]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:32.096843 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571d6b5b23424ea29dbe787584525efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:32.113833 12463 tablet_bootstrap.cc:492] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:32.115284 12462 tablet_bootstrap.cc:492] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:32.119592 12461 tablet_bootstrap.cc:492] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:32.119926 12463 tablet_bootstrap.cc:654] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.121433 12463 log.cc:824] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:32.123065 12461 tablet_bootstrap.cc:654] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.124327 12461 log.cc:824] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:32.125489 12462 tablet_bootstrap.cc:654] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.126649 12463 tablet_bootstrap.cc:492] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:32.126830 12462 log.cc:824] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:32.126935 12463 ts_tablet_manager.cc:1367] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240501 14:01:32.126914 12465 tablet_bootstrap.cc:492] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:32.128091 12461 tablet_bootstrap.cc:492] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:32.128408 12461 ts_tablet_manager.cc:1367] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 14:01:32.130724 12465 tablet_bootstrap.cc:654] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.132004 12465 log.cc:824] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8: Log is configured to *not* fsync() on all Append() calls
I20240501 14:01:32.135877 12463 raft_consensus.cc:348] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.136417 12463 raft_consensus.cc:374] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.136497 12462 tablet_bootstrap.cc:492] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:32.136514 12463 raft_consensus.cc:729] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.136809 12462 ts_tablet_manager.cc:1367] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.007s
I20240501 14:01:32.137414 12463 consensus_queue.cc:260] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.139158 12461 raft_consensus.cc:348] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.139614 12461 raft_consensus.cc:374] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.139722 12461 raft_consensus.cc:729] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.140571 12461 consensus_queue.cc:260] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.140909 12463 ts_tablet_manager.cc:1398] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20240501 14:01:32.141743 12463 tablet_bootstrap.cc:492] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:32.141873 12465 tablet_bootstrap.cc:492] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:32.142237 12465 ts_tablet_manager.cc:1367] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240501 14:01:32.143517 12463 tablet_bootstrap.cc:654] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.149142 12463 tablet_bootstrap.cc:492] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:32.149155 12461 ts_tablet_manager.cc:1398] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20240501 14:01:32.149317 12463 ts_tablet_manager.cc:1367] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240501 14:01:32.149855 12461 tablet_bootstrap.cc:492] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:32.150314 12463 raft_consensus.cc:348] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.150570 12463 raft_consensus.cc:374] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.150671 12463 raft_consensus.cc:729] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.150950 12463 consensus_queue.cc:260] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.151597 12463 ts_tablet_manager.cc:1398] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:01:32.151643 12461 tablet_bootstrap.cc:654] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.151975 12465 raft_consensus.cc:348] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.153136 12465 raft_consensus.cc:374] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.153865 12465 raft_consensus.cc:729] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.154851 12465 consensus_queue.cc:260] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.156349 12462 raft_consensus.cc:348] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.156904 12462 raft_consensus.cc:374] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.157016 12462 raft_consensus.cc:729] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.157832 12462 consensus_queue.cc:260] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.160574 12465 ts_tablet_manager.cc:1398] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.018s user 0.012s sys 0.002s
I20240501 14:01:32.160584 12462 ts_tablet_manager.cc:1398] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.024s user 0.011s sys 0.004s
I20240501 14:01:32.161082 12465 tablet_bootstrap.cc:492] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:32.161116 12462 tablet_bootstrap.cc:492] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:32.162902 12462 tablet_bootstrap.cc:654] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.163225 12465 tablet_bootstrap.cc:654] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:32.165239 12461 tablet_bootstrap.cc:492] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:32.165422 12461 ts_tablet_manager.cc:1367] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20240501 14:01:32.166541 12461 raft_consensus.cc:348] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.166766 12461 raft_consensus.cc:374] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.166868 12461 raft_consensus.cc:729] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.167110 12461 consensus_queue.cc:260] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.167755 12461 ts_tablet_manager.cc:1398] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:32.169638 12462 tablet_bootstrap.cc:492] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:32.169806 12465 tablet_bootstrap.cc:492] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:32.169966 12465 ts_tablet_manager.cc:1367] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 14:01:32.169967 12462 ts_tablet_manager.cc:1367] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 14:01:32.170993 12462 raft_consensus.cc:348] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.170993 12465 raft_consensus.cc:348] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.171211 12465 raft_consensus.cc:374] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.171209 12462 raft_consensus.cc:374] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.171304 12462 raft_consensus.cc:729] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.171314 12465 raft_consensus.cc:729] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.171535 12465 consensus_queue.cc:260] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.171535 12462 consensus_queue.cc:260] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.172034 12465 ts_tablet_manager.cc:1398] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:32.172305 12465 tablet_bootstrap.cc:492] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:32.173128 12462 ts_tablet_manager.cc:1398] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:32.173899 12465 tablet_bootstrap.cc:654] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
W20240501 14:01:32.175274 11913 tablet.cc:2369] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:01:32.176229 12465 tablet_bootstrap.cc:492] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:32.176355 12465 ts_tablet_manager.cc:1367] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:32.176954 12465 raft_consensus.cc:348] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.177160 12465 raft_consensus.cc:374] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:32.177232 12465 raft_consensus.cc:729] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:32.177448 12465 consensus_queue.cc:260] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.177985 12465 ts_tablet_manager.cc:1398] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:32.190058 12474 raft_consensus.cc:482] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:32.190317 12474 raft_consensus.cc:504] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.192135 12474 leader_election.cc:288] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:32.199101 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:32.199101 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:32.199597 11476 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:32.199596 11737 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:32.200206 11540 leader_election.cc:302] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:32.200560 12474 raft_consensus.cc:2781] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:32.200692 12474 raft_consensus.cc:482] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:32.200774 12474 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.202931 12474 raft_consensus.cc:504] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.203541 12474 leader_election.cc:288] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:32.203967 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:32.204053 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:32.204154 11737 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.204243 11476 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.206205 11737 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:32.206240 11476 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:32.206666 11543 leader_election.cc:302] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:32.207051 12474 raft_consensus.cc:2781] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:32.207515 12474 raft_consensus.cc:686] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:32.208026 12474 consensus_queue.cc:237] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:32.212129 11335 catalog_manager.cc:5484] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
W20240501 14:01:32.226227 11783 tablet.cc:2369] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:01:32.229110 12469 raft_consensus.cc:482] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:32.229317 12469 raft_consensus.cc:504] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.230893 12474 raft_consensus.cc:482] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:32.231020 12469 leader_election.cc:288] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:32.231118 12474 raft_consensus.cc:504] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.231822 12474 leader_election.cc:288] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:32.233100 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:32.233378 11476 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:32.233861 11543 leader_election.cc:302] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:32.234400 12474 raft_consensus.cc:2781] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:32.234570 12474 raft_consensus.cc:482] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:32.234715 12474 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.237799 12474 raft_consensus.cc:504] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.239151 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:32.239424 11476 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.243897 12470 raft_consensus.cc:482] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:32.244189 12470 raft_consensus.cc:504] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.245558 11476 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:32.246112 11543 leader_election.cc:302] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:32.246416 12480 raft_consensus.cc:2781] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:32.246420 12470 leader_election.cc:288] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:32.246917 12474 leader_election.cc:288] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:32.248664 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:32.249253 11867 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:32.249981 11673 leader_election.cc:302] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:32.250533 12469 raft_consensus.cc:2781] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:32.250670 12469 raft_consensus.cc:482] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:32.250762 12469 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.253643 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:32.253885 11476 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:32.253949 12469 raft_consensus.cc:504] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.254747 12469 leader_election.cc:288] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:32.255293 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:32.255312 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:32.255525 11476 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.255517 11867 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.256615 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:32.257025 11475 raft_consensus.cc:2379] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e in current term 1: Already voted for candidate cf637c0b31e34127b7b2599a8c2763a6 in this term.
I20240501 14:01:32.258296 11476 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:32.258790 11673 leader_election.cc:302] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:32.259022 12480 raft_consensus.cc:686] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:32.259352 12480 consensus_queue.cc:237] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:32.259483 11867 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:32.259828 12469 raft_consensus.cc:2781] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:32.261318 12469 raft_consensus.cc:686] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:32.261909 12469 consensus_queue.cc:237] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:32.262638 11335 catalog_manager.cc:5484] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:32.264021 11653 tablet.cc:2369] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:01:32.268891 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:32.269179 11867 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:32.269217 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:32.269491 11866 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:32.269901 11335 catalog_manager.cc:5484] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 1 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:32.271905 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:32.272147 11866 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:32.272976 11802 leader_election.cc:302] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:32.273416 12470 raft_consensus.cc:2726] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 14:01:32.303660 11522 tablet.cc:2369] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:01:32.618170 12492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:32.618486 12492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:32.626080 12492 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:01:32.677276 12480 consensus_queue.cc:1035] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.684482 12469 consensus_queue.cc:1035] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.701395 12469 consensus_queue.cc:1035] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.705849 12480 consensus_queue.cc:1035] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.766448 11476 raft_consensus.cc:1261] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Refusing update from remote peer cf637c0b31e34127b7b2599a8c2763a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:32.767302 12474 consensus_queue.cc:1035] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.781014 11737 raft_consensus.cc:1261] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Refusing update from remote peer cf637c0b31e34127b7b2599a8c2763a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:32.782837 12480 consensus_queue.cc:1035] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:32.811553 12533 mvcc.cc:205] Tried to move back new op lower bound from 7022887291960033280 to 7022887289687834624. Current Snapshot: MvccSnapshot[applied={T|T < 7022887291960033280}]
I20240501 14:01:32.822279 12528 mvcc.cc:205] Tried to move back new op lower bound from 7022887291960033280 to 7022887289687834624. Current Snapshot: MvccSnapshot[applied={T|T < 7022887291960033280}]
INSERT report
rows total: 2000
time total: 212.787 ms
time per row: 0.106394 ms
I20240501 14:01:32.992559 11333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:33.001300 11717 tablet_service.cc:1467] Processing CreateTablet for tablet a7f0f7056f9449dfb41c885d216102cf (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.001502 11847 tablet_service.cc:1467] Processing CreateTablet for tablet a7f0f7056f9449dfb41c885d216102cf (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.001822 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f0f7056f9449dfb41c885d216102cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.001924 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f0f7056f9449dfb41c885d216102cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.002032 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 4e510660118b4d71a1835c6ffa5a02ed (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.002403 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e510660118b4d71a1835c6ffa5a02ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.003366 11456 tablet_service.cc:1467] Processing CreateTablet for tablet 91002b1d230d442e91be4067b39c8acb (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.003772 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91002b1d230d442e91be4067b39c8acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.005476 11586 tablet_service.cc:1467] Processing CreateTablet for tablet a7f0f7056f9449dfb41c885d216102cf (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.005868 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f0f7056f9449dfb41c885d216102cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.007004 11587 tablet_service.cc:1467] Processing CreateTablet for tablet 4e510660118b4d71a1835c6ffa5a02ed (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.007398 11587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e510660118b4d71a1835c6ffa5a02ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.008982 11715 tablet_service.cc:1467] Processing CreateTablet for tablet 91002b1d230d442e91be4067b39c8acb (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.009426 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91002b1d230d442e91be4067b39c8acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.010977 11846 tablet_service.cc:1467] Processing CreateTablet for tablet 4e510660118b4d71a1835c6ffa5a02ed (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.011459 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e510660118b4d71a1835c6ffa5a02ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.011839 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 91002b1d230d442e91be4067b39c8acb (DEFAULT_TABLE table=rebalance_test_table_1 [id=ac12d8c84bfa417c9a5013d02eaff23b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:33.012228 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91002b1d230d442e91be4067b39c8acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:33.030735 12555 tablet_bootstrap.cc:492] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:33.031423 12556 tablet_bootstrap.cc:492] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:33.033293 12556 tablet_bootstrap.cc:654] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.033344 12555 tablet_bootstrap.cc:654] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.035548 12553 tablet_bootstrap.cc:492] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:33.039129 12553 tablet_bootstrap.cc:654] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.040871 12556 tablet_bootstrap.cc:492] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:33.041069 12556 ts_tablet_manager.cc:1367] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 14:01:33.042037 12556 raft_consensus.cc:348] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.042320 12556 raft_consensus.cc:374] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.042407 12556 raft_consensus.cc:729] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.042687 12556 consensus_queue.cc:260] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.043375 12556 ts_tablet_manager.cc:1398] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:33.043687 12556 tablet_bootstrap.cc:492] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:33.045490 12556 tablet_bootstrap.cc:654] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.047565 12553 tablet_bootstrap.cc:492] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:33.047698 12556 tablet_bootstrap.cc:492] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:33.047842 12553 ts_tablet_manager.cc:1367] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:01:33.048034 12556 ts_tablet_manager.cc:1367] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 14:01:33.048940 12556 raft_consensus.cc:348] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.049219 12556 raft_consensus.cc:374] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.049304 12556 raft_consensus.cc:729] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.049604 12556 consensus_queue.cc:260] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.050226 12556 ts_tablet_manager.cc:1398] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:33.051249 12556 tablet_bootstrap.cc:492] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:33.053010 12556 tablet_bootstrap.cc:654] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.054351 12553 raft_consensus.cc:348] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.054725 12553 raft_consensus.cc:374] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.054888 12553 raft_consensus.cc:729] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.055234 12553 consensus_queue.cc:260] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.055923 12553 ts_tablet_manager.cc:1398] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:33.056334 12553 tablet_bootstrap.cc:492] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:33.055549 12556 tablet_bootstrap.cc:492] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:33.057857 12556 ts_tablet_manager.cc:1367] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240501 14:01:33.058101 12553 tablet_bootstrap.cc:654] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.058750 12556 raft_consensus.cc:348] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.058977 12556 raft_consensus.cc:374] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.059070 12556 raft_consensus.cc:729] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.059324 12556 consensus_queue.cc:260] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.060029 12556 ts_tablet_manager.cc:1398] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:33.064144 12554 tablet_bootstrap.cc:492] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:33.066349 12554 tablet_bootstrap.cc:654] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.068305 12555 tablet_bootstrap.cc:492] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:33.068565 12555 ts_tablet_manager.cc:1367] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.012s
I20240501 14:01:33.069545 12555 raft_consensus.cc:348] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.069869 12555 raft_consensus.cc:374] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.070000 12555 raft_consensus.cc:729] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.070369 12555 consensus_queue.cc:260] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.071208 12555 ts_tablet_manager.cc:1398] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:33.073426 12555 tablet_bootstrap.cc:492] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:33.076617 12553 tablet_bootstrap.cc:492] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:33.076797 12553 ts_tablet_manager.cc:1367] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240501 14:01:33.077394 12555 tablet_bootstrap.cc:654] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.077544 12553 raft_consensus.cc:348] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.078192 12553 raft_consensus.cc:374] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.078281 12553 raft_consensus.cc:729] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.078572 12553 consensus_queue.cc:260] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.079094 12553 ts_tablet_manager.cc:1398] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:33.079442 12553 tablet_bootstrap.cc:492] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:33.080957 12553 tablet_bootstrap.cc:654] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:33.085888 12485 raft_consensus.cc:482] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:33.086108 12485 raft_consensus.cc:504] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.087060 12485 leader_election.cc:288] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:33.087546 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:33.087867 11867 raft_consensus.cc:2454] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:33.087925 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:33.088202 11737 raft_consensus.cc:2454] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:33.088356 11543 leader_election.cc:302] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:33.088711 12485 raft_consensus.cc:2781] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:33.088829 12485 raft_consensus.cc:482] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:33.088913 12485 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.089058 12554 tablet_bootstrap.cc:492] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:33.089255 12554 ts_tablet_manager.cc:1367] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.007s
I20240501 14:01:33.090174 12554 raft_consensus.cc:348] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.090417 12554 raft_consensus.cc:374] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.090559 12554 raft_consensus.cc:729] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.090842 12554 consensus_queue.cc:260] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.092063 12485 raft_consensus.cc:504] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.092907 12554 ts_tablet_manager.cc:1398] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:01:33.094333 12485 leader_election.cc:288] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:33.094560 12553 tablet_bootstrap.cc:492] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:33.094653 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:33.094731 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:33.094851 11737 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.094956 11867 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.095388 12555 tablet_bootstrap.cc:492] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:33.095558 12555 ts_tablet_manager.cc:1367] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.015s
I20240501 14:01:33.096365 12555 raft_consensus.cc:348] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.096583 12555 raft_consensus.cc:374] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.096663 12555 raft_consensus.cc:729] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.096910 12555 consensus_queue.cc:260] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.097436 12555 ts_tablet_manager.cc:1398] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:33.097458 11737 raft_consensus.cc:2454] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:33.097438 11867 raft_consensus.cc:2454] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:33.094724 12553 ts_tablet_manager.cc:1367] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.009s
I20240501 14:01:33.097965 11540 leader_election.cc:302] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:33.098434 12485 raft_consensus.cc:2781] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:33.098420 12553 raft_consensus.cc:348] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.098620 12485 raft_consensus.cc:686] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:33.098663 12553 raft_consensus.cc:374] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:33.098764 12553 raft_consensus.cc:729] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:33.098893 12485 consensus_queue.cc:237] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.098985 12553 consensus_queue.cc:260] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.099475 12553 ts_tablet_manager.cc:1398] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:33.102923 11333 catalog_manager.cc:5484] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:33.136058 12485 raft_consensus.cc:482] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:33.136312 12485 raft_consensus.cc:504] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.137166 12485 leader_election.cc:288] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:33.137814 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:33.137804 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:33.138051 11475 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:33.138072 11737 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:33.138548 11543 leader_election.cc:302] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:33.138882 12485 raft_consensus.cc:2781] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:33.138993 12485 raft_consensus.cc:482] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:33.139099 12485 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.141896 12485 raft_consensus.cc:504] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.142710 12485 leader_election.cc:288] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:33.143237 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:33.143270 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:33.143500 11737 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.143508 11475 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.146155 11737 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:33.146302 11475 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:33.146612 11540 leader_election.cc:302] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:33.146939 12485 raft_consensus.cc:2781] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:33.147126 12485 raft_consensus.cc:686] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:33.147411 12485 consensus_queue.cc:237] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:33.150283 11333 catalog_manager.cc:5484] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:33.356681 12524 raft_consensus.cc:482] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:33.356894 12524 raft_consensus.cc:504] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.357699 12524 leader_election.cc:288] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:33.358280 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:33.358599 11867 raft_consensus.cc:2454] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:33.359047 11673 leader_election.cc:302] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:33.359345 12524 raft_consensus.cc:2781] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:33.359474 12524 raft_consensus.cc:482] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:33.359583 12524 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.361891 12524 raft_consensus.cc:504] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.362795 12524 leader_election.cc:288] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:33.363176 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:33.363406 11867 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.363936 12485 raft_consensus.cc:482] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:33.364123 12485 raft_consensus.cc:504] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.364979 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:33.365129 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:33.365267 11607 raft_consensus.cc:2454] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:33.365674 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:33.365936 11737 raft_consensus.cc:2379] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 in current term 1: Already voted for candidate cd59a6f0c91a434d813bbf69a0b66b6b in this term.
I20240501 14:01:33.365945 11867 raft_consensus.cc:2454] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:33.366432 12485 leader_election.cc:288] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:33.366389 11673 leader_election.cc:302] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters: cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:33.366533 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:33.366724 12524 raft_consensus.cc:2781] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:33.366807 11867 raft_consensus.cc:2379] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 in current term 1: Already voted for candidate cd59a6f0c91a434d813bbf69a0b66b6b in this term.
I20240501 14:01:33.366868 12524 raft_consensus.cc:686] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:33.367128 12524 consensus_queue.cc:237] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:33.367262 11543 leader_election.cc:302] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf637c0b31e34127b7b2599a8c2763a6; no voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:33.367501 12485 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:33.369890 12485 raft_consensus.cc:2726] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:33.369877 11333 catalog_manager.cc:5484] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 1 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:33.481060 12485 consensus_queue.cc:1035] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:33.500087 12480 consensus_queue.cc:1035] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:33.723026 12571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:33.723392 12571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:33.731129 12571 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:33.732636 12480 consensus_queue.cc:1035] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:33.738760 12575 consensus_queue.cc:1035] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 14:01:33.859468 12524 consensus_queue.cc:1035] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:33.902902 11607 raft_consensus.cc:1261] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:33.913523 12618 consensus_queue.cc:1035] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 242.182 ms
time per row: 0.121091 ms
I20240501 14:01:34.125087 11335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:34.133615 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc63424584f4016967b1e2cf0352ee4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.134088 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc63424584f4016967b1e2cf0352ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.134081 11456 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc63424584f4016967b1e2cf0352ee4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.134541 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc63424584f4016967b1e2cf0352ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.134932 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 8d00ce95a04f49dfa3df17ee8998955c (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.135341 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d00ce95a04f49dfa3df17ee8998955c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.135341 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 8d00ce95a04f49dfa3df17ee8998955c (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.135726 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d00ce95a04f49dfa3df17ee8998955c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.139585 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 8d00ce95a04f49dfa3df17ee8998955c (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.139976 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d00ce95a04f49dfa3df17ee8998955c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.140844 11586 tablet_service.cc:1467] Processing CreateTablet for tablet 2563e19394b1476d89a4bafa24aa90c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.141224 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2563e19394b1476d89a4bafa24aa90c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.141449 11846 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc63424584f4016967b1e2cf0352ee4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.141804 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc63424584f4016967b1e2cf0352ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.143503 11454 tablet_service.cc:1467] Processing CreateTablet for tablet 2563e19394b1476d89a4bafa24aa90c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.143936 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2563e19394b1476d89a4bafa24aa90c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.147475 11845 tablet_service.cc:1467] Processing CreateTablet for tablet 2563e19394b1476d89a4bafa24aa90c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=44bfe7ae186e4f3f8c19cfb1c8112133]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:34.147986 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2563e19394b1476d89a4bafa24aa90c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:34.151515 12639 tablet_bootstrap.cc:492] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:34.153985 12639 tablet_bootstrap.cc:654] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.159094 12638 tablet_bootstrap.cc:492] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:34.159931 12639 tablet_bootstrap.cc:492] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:34.160153 12639 ts_tablet_manager.cc:1367] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 14:01:34.161244 12639 raft_consensus.cc:348] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.161491 12639 raft_consensus.cc:374] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.161607 12639 raft_consensus.cc:729] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.161844 12639 consensus_queue.cc:260] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.162632 12639 ts_tablet_manager.cc:1398] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:34.162952 12639 tablet_bootstrap.cc:492] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:34.164651 12639 tablet_bootstrap.cc:654] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.165508 12638 tablet_bootstrap.cc:654] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.169566 12637 tablet_bootstrap.cc:492] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:34.171847 12641 tablet_bootstrap.cc:492] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:34.172906 12637 tablet_bootstrap.cc:654] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.174188 12638 tablet_bootstrap.cc:492] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:34.174422 12638 ts_tablet_manager.cc:1367] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240501 14:01:34.174444 12639 tablet_bootstrap.cc:492] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:34.174654 12639 ts_tablet_manager.cc:1367] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20240501 14:01:34.175118 12641 tablet_bootstrap.cc:654] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.175433 12638 raft_consensus.cc:348] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.175550 12639 raft_consensus.cc:348] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.175772 12638 raft_consensus.cc:374] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.175855 12639 raft_consensus.cc:374] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.175912 12638 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.175988 12639 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.176323 12638 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.176360 12639 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.177073 12638 ts_tablet_manager.cc:1398] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:34.177417 12638 tablet_bootstrap.cc:492] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:34.177791 12639 ts_tablet_manager.cc:1398] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:34.178141 12639 tablet_bootstrap.cc:492] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:34.179212 12638 tablet_bootstrap.cc:654] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.179744 12639 tablet_bootstrap.cc:654] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.187492 12638 tablet_bootstrap.cc:492] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:34.187821 12638 ts_tablet_manager.cc:1367] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 14:01:34.188711 12638 raft_consensus.cc:348] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.189173 12638 raft_consensus.cc:374] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.189301 12638 raft_consensus.cc:729] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.189640 12638 consensus_queue.cc:260] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.190430 12638 ts_tablet_manager.cc:1398] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:34.190853 12638 tablet_bootstrap.cc:492] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:34.192929 12638 tablet_bootstrap.cc:654] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.198612 12641 tablet_bootstrap.cc:492] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:34.198899 12641 ts_tablet_manager.cc:1367] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20240501 14:01:34.199891 12641 raft_consensus.cc:348] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.200239 12641 raft_consensus.cc:374] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.200403 12641 raft_consensus.cc:729] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.200788 12641 consensus_queue.cc:260] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.201601 12641 ts_tablet_manager.cc:1398] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:34.201618 12638 tablet_bootstrap.cc:492] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:34.201769 12638 ts_tablet_manager.cc:1367] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240501 14:01:34.202634 12638 raft_consensus.cc:348] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.202909 12638 raft_consensus.cc:374] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.203042 12638 raft_consensus.cc:729] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.203390 12639 tablet_bootstrap.cc:492] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:34.203368 12638 consensus_queue.cc:260] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.203558 12637 tablet_bootstrap.cc:492] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:34.203591 12639 ts_tablet_manager.cc:1367] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.008s
I20240501 14:01:34.204080 12638 ts_tablet_manager.cc:1398] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:34.203745 12637 ts_tablet_manager.cc:1367] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.016s
I20240501 14:01:34.204339 12639 raft_consensus.cc:348] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.204607 12639 raft_consensus.cc:374] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.204744 12639 raft_consensus.cc:729] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.205077 12639 consensus_queue.cc:260] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.205154 12637 raft_consensus.cc:348] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.205405 12637 raft_consensus.cc:374] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.205515 12637 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.205672 12639 ts_tablet_manager.cc:1398] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:34.205772 12637 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.206331 12637 ts_tablet_manager.cc:1398] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:34.206660 12637 tablet_bootstrap.cc:492] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:34.206939 12470 raft_consensus.cc:482] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:34.207078 12470 raft_consensus.cc:504] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.207803 12470 leader_election.cc:288] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:34.208215 12637 tablet_bootstrap.cc:654] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:34.209275 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:34.209539 11475 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:34.210105 11803 leader_election.cc:302] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:34.210726 12470 raft_consensus.cc:2781] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:34.210851 12470 raft_consensus.cc:482] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:34.210942 12470 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.213351 12637 tablet_bootstrap.cc:492] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:34.213563 12637 ts_tablet_manager.cc:1367] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20240501 14:01:34.214211 12470 raft_consensus.cc:504] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.214428 12637 raft_consensus.cc:348] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.214685 12637 raft_consensus.cc:374] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:34.214766 12637 raft_consensus.cc:729] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:34.215052 12637 consensus_queue.cc:260] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.215400 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:34.215632 11475 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.216150 12637 ts_tablet_manager.cc:1398] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:34.218189 11475 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:34.218662 12470 leader_election.cc:288] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:34.218662 11803 leader_election.cc:302] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:34.219085 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:34.219322 11737 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:34.219352 12651 raft_consensus.cc:2781] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:34.219480 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:34.219522 12651 raft_consensus.cc:686] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:34.219635 11736 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.220054 12651 consensus_queue.cc:237] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:34.221884 11736 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:34.223714 11335 catalog_manager.cc:5484] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e reported cstate change: term changed from 0 to 1, leader changed from <none> to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 1 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:34.238489 12575 raft_consensus.cc:482] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:34.238727 12575 raft_consensus.cc:504] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.239418 12575 leader_election.cc:288] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:34.239933 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:34.240013 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:34.240139 11475 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:34.240267 11866 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:34.240557 11543 leader_election.cc:302] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:34.240839 12575 raft_consensus.cc:2781] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:34.240927 12575 raft_consensus.cc:482] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:34.241021 12575 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.242978 12575 raft_consensus.cc:504] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.243544 12575 leader_election.cc:288] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:34.244099 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:34.244117 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:34.244277 11475 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.244313 11866 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.246142 11475 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:34.246142 11866 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:34.246569 11543 leader_election.cc:302] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:34.246835 12575 raft_consensus.cc:2781] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:34.247012 12575 raft_consensus.cc:686] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:34.247290 12575 consensus_queue.cc:237] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.250041 11335 catalog_manager.cc:5484] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:34.279911 12470 raft_consensus.cc:482] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:34.280113 12470 raft_consensus.cc:504] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.280825 12470 leader_election.cc:288] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:34.281343 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:34.281414 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:34.281553 11607 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:34.281653 11475 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:34.281972 11802 leader_election.cc:302] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:34.282359 12470 raft_consensus.cc:2781] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:34.282465 12470 raft_consensus.cc:482] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:34.282569 12470 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.284409 12470 raft_consensus.cc:504] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.284932 12470 leader_election.cc:288] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:34.285470 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:34.285451 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:34.285665 11607 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.285686 11475 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:34.288465 11475 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:34.288482 11607 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:34.288929 11803 leader_election.cc:302] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:34.289279 12470 raft_consensus.cc:2781] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:34.289440 12470 raft_consensus.cc:686] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:34.289709 12470 consensus_queue.cc:237] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:34.292618 11335 catalog_manager.cc:5484] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 0 to 1, leader changed from <none> to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 1 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
W20240501 14:01:34.642019 12655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:34.642396 12655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:34.649884 12655 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:34.677608 12470 consensus_queue.cc:1035] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 14:01:34.685592 12673 consensus_queue.cc:1035] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.706740 12575 consensus_queue.cc:1035] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.712752 12612 consensus_queue.cc:1035] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.715288 12470 consensus_queue.cc:1035] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:34.720273 12651 consensus_queue.cc:1035] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:34.915997 11803 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.17.1:44355, user_credentials={real_user=slave}} blocked reactor thread for 48635.1us
INSERT report
rows total: 2000
time total: 224.17 ms
time per row: 0.112085 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:01:35.897651 11326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:35.906829 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 2a35808076b34ec49f8897a477fc0b3d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.907043 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 2a35808076b34ec49f8897a477fc0b3d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.907272 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a35808076b34ec49f8897a477fc0b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.907474 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a35808076b34ec49f8897a477fc0b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.908648 11845 tablet_service.cc:1467] Processing CreateTablet for tablet 387b254002554b1cb0e41ea5f9092a4d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.908735 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 825b8c2dd4504ce7acf0b8f8218ac9ca (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.909055 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387b254002554b1cb0e41ea5f9092a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.909353 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825b8c2dd4504ce7acf0b8f8218ac9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.910892 11454 tablet_service.cc:1467] Processing CreateTablet for tablet 2a35808076b34ec49f8897a477fc0b3d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.911305 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a35808076b34ec49f8897a477fc0b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.911444 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 387b254002554b1cb0e41ea5f9092a4d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.911808 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387b254002554b1cb0e41ea5f9092a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.913322 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 387b254002554b1cb0e41ea5f9092a4d (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.913348 11586 tablet_service.cc:1467] Processing CreateTablet for tablet 825b8c2dd4504ce7acf0b8f8218ac9ca (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.913709 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387b254002554b1cb0e41ea5f9092a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.913712 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825b8c2dd4504ce7acf0b8f8218ac9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.918043 11715 tablet_service.cc:1467] Processing CreateTablet for tablet 825b8c2dd4504ce7acf0b8f8218ac9ca (DEFAULT_TABLE table=rebalance_test_table_3 [id=a233f513eeff4262b7f928754e98a32e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:35.918552 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825b8c2dd4504ce7acf0b8f8218ac9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:35.923826 12717 tablet_bootstrap.cc:492] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:35.925607 12717 tablet_bootstrap.cc:654] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.933499 12718 tablet_bootstrap.cc:492] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:35.935314 12718 tablet_bootstrap.cc:654] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.943814 12721 tablet_bootstrap.cc:492] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:35.943915 12718 tablet_bootstrap.cc:492] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:35.944069 12718 ts_tablet_manager.cc:1367] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 14:01:35.944917 12718 raft_consensus.cc:348] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:35.945209 12718 raft_consensus.cc:374] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.945309 12718 raft_consensus.cc:729] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.945441 12719 tablet_bootstrap.cc:492] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:35.945593 12718 consensus_queue.cc:260] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:35.946185 12721 tablet_bootstrap.cc:654] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.947135 12718 ts_tablet_manager.cc:1398] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:35.947278 12719 tablet_bootstrap.cc:654] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.947438 12718 tablet_bootstrap.cc:492] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:35.949013 12718 tablet_bootstrap.cc:654] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.949473 12717 tablet_bootstrap.cc:492] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:35.949633 12717 ts_tablet_manager.cc:1367] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.021s
I20240501 14:01:35.950415 12717 raft_consensus.cc:348] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.950652 12717 raft_consensus.cc:374] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.950743 12717 raft_consensus.cc:729] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.951015 12717 consensus_queue.cc:260] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.951812 12717 ts_tablet_manager.cc:1398] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:35.952080 12718 tablet_bootstrap.cc:492] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:35.952342 12718 ts_tablet_manager.cc:1367] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:01:35.952363 12717 tablet_bootstrap.cc:492] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:35.953143 12718 raft_consensus.cc:348] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.953429 12718 raft_consensus.cc:374] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.953547 12718 raft_consensus.cc:729] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.953855 12718 consensus_queue.cc:260] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.954396 12718 ts_tablet_manager.cc:1398] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:35.954437 12717 tablet_bootstrap.cc:654] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.954761 12718 tablet_bootstrap.cc:492] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:35.955950 12719 tablet_bootstrap.cc:492] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:35.956108 12719 ts_tablet_manager.cc:1367] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 14:01:35.956300 12718 tablet_bootstrap.cc:654] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.956840 12719 raft_consensus.cc:348] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.957032 12719 raft_consensus.cc:374] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.957118 12719 raft_consensus.cc:729] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.957353 12719 consensus_queue.cc:260] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.960352 12721 tablet_bootstrap.cc:492] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:35.960564 12721 ts_tablet_manager.cc:1367] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:01:35.961360 12721 raft_consensus.cc:348] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:35.961633 12721 raft_consensus.cc:374] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.961759 12721 raft_consensus.cc:729] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.962047 12721 consensus_queue.cc:260] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:35.962752 12721 ts_tablet_manager.cc:1398] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:35.964602 12721 tablet_bootstrap.cc:492] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:35.965384 12719 ts_tablet_manager.cc:1398] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240501 14:01:35.965757 12719 tablet_bootstrap.cc:492] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:35.967443 12719 tablet_bootstrap.cc:654] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.969288 12717 tablet_bootstrap.cc:492] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:35.969357 12721 tablet_bootstrap.cc:654] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:35.969542 12717 ts_tablet_manager.cc:1367] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240501 14:01:35.970356 12717 raft_consensus.cc:348] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:35.970583 12717 raft_consensus.cc:374] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.970654 12717 raft_consensus.cc:729] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.970894 12717 consensus_queue.cc:260] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:35.971534 12717 ts_tablet_manager.cc:1398] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:35.972012 12719 tablet_bootstrap.cc:492] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:35.972167 12718 tablet_bootstrap.cc:492] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:35.972173 12719 ts_tablet_manager.cc:1367] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 14:01:35.972316 12721 tablet_bootstrap.cc:492] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:35.972321 12718 ts_tablet_manager.cc:1367] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20240501 14:01:35.972458 12721 ts_tablet_manager.cc:1367] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240501 14:01:35.972889 12719 raft_consensus.cc:348] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.973095 12719 raft_consensus.cc:374] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.973219 12719 raft_consensus.cc:729] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.973312 12718 raft_consensus.cc:348] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.973512 12718 raft_consensus.cc:374] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.973601 12718 raft_consensus.cc:729] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.973495 12719 consensus_queue.cc:260] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.973812 12718 consensus_queue.cc:260] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.973208 12721 raft_consensus.cc:348] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.974045 12719 ts_tablet_manager.cc:1398] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:35.974450 12718 ts_tablet_manager.cc:1398] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:35.974079 12721 raft_consensus.cc:374] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:35.975553 12721 raft_consensus.cc:729] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:35.976066 12721 consensus_queue.cc:260] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:35.976609 12721 ts_tablet_manager.cc:1398] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:01:36.000485 12470 raft_consensus.cc:482] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:36.000712 12470 raft_consensus.cc:504] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:36.001372 12470 leader_election.cc:288] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:36.001930 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:36.001957 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:36.002174 11736 raft_consensus.cc:2454] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:36.002184 11475 raft_consensus.cc:2454] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:36.002703 11803 leader_election.cc:302] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:36.002991 12470 raft_consensus.cc:2781] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:36.003083 12470 raft_consensus.cc:482] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:36.003176 12470 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.005117 12470 raft_consensus.cc:504] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:36.005831 12470 leader_election.cc:288] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:36.006217 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:36.006301 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:36.006438 11736 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.006482 11475 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.009210 11736 raft_consensus.cc:2454] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:36.009225 11475 raft_consensus.cc:2454] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:36.009733 11800 leader_election.cc:302] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:36.010054 12470 raft_consensus.cc:2781] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:36.010524 12470 raft_consensus.cc:686] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:36.010978 12470 consensus_queue.cc:237] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:36.013770 11335 catalog_manager.cc:5484] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e reported cstate change: term changed from 0 to 1, leader changed from <none> to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 1 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:36.222622 12700 raft_consensus.cc:482] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:36.222831 12700 raft_consensus.cc:504] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:36.223508 12700 leader_election.cc:288] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:36.224017 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:36.224040 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:36.224269 11866 raft_consensus.cc:2454] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:36.224264 11736 raft_consensus.cc:2454] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:36.224694 11543 leader_election.cc:302] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:36.224960 12700 raft_consensus.cc:2781] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:36.225070 12700 raft_consensus.cc:482] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:36.225152 12700 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.227030 12700 raft_consensus.cc:504] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:36.227593 12700 leader_election.cc:288] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:36.228142 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:36.228142 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:36.228348 11736 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.228348 11866 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.231097 11736 raft_consensus.cc:2454] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:36.231096 11866 raft_consensus.cc:2454] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:36.231561 11543 leader_election.cc:302] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:36.231875 12700 raft_consensus.cc:2781] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:36.232271 12700 raft_consensus.cc:686] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:36.232571 12700 consensus_queue.cc:237] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:36.235141 11335 catalog_manager.cc:5484] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:36.318989 12700 raft_consensus.cc:482] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:36.319208 12700 raft_consensus.cc:504] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:36.319864 12700 leader_election.cc:288] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:36.320425 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:36.320436 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:36.320655 11475 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:36.320653 11736 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:36.321043 11540 leader_election.cc:302] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:36.321331 12700 raft_consensus.cc:2781] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:36.321414 12700 raft_consensus.cc:482] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:36.321509 12700 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.323432 12700 raft_consensus.cc:504] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:36.324019 12700 leader_election.cc:288] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:36.324506 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:36.324506 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:36.324700 11475 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.324697 11736 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:36.326529 11475 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:36.326546 11736 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:36.326949 11543 leader_election.cc:302] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:36.327255 12700 raft_consensus.cc:2781] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:36.327417 12700 raft_consensus.cc:686] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:36.327684 12700 consensus_queue.cc:237] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:36.330288 11335 catalog_manager.cc:5484] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:36.467684 12470 consensus_queue.cc:1035] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:36.475158 12731 consensus_queue.cc:1035] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:36.657842 12700 consensus_queue.cc:1035] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:36.667510 12744 consensus_queue.cc:1035] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:36.726123 12736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:36.726461 12736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:36.733906 12736 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:01:36.771304 12745 consensus_queue.cc:1035] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:36.776211 12700 consensus_queue.cc:1035] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.233 ms
time per row: 0.117616 ms
I20240501 14:01:37.123723 11326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:37.132604 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb2420c2ca4eefa11a882651d7f9ac (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.132714 11586 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb2420c2ca4eefa11a882651d7f9ac (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.133098 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb2420c2ca4eefa11a882651d7f9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.133039 11454 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb2420c2ca4eefa11a882651d7f9ac (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.133150 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb2420c2ca4eefa11a882651d7f9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.134742 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 901d22dfa4fd4cf2b574875ef4f8135b (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.135248 11715 tablet_service.cc:1467] Processing CreateTablet for tablet 120276baa7874d5aa0351a053598ab0a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.135421 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 901d22dfa4fd4cf2b574875ef4f8135b (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.135692 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120276baa7874d5aa0351a053598ab0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.136243 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 901d22dfa4fd4cf2b574875ef4f8135b (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.136610 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 901d22dfa4fd4cf2b574875ef4f8135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.138054 11456 tablet_service.cc:1467] Processing CreateTablet for tablet 120276baa7874d5aa0351a053598ab0a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.138552 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 901d22dfa4fd4cf2b574875ef4f8135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.139808 11845 tablet_service.cc:1467] Processing CreateTablet for tablet 120276baa7874d5aa0351a053598ab0a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9b3a3e639f624fb9bccd450d91e9892f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:37.140246 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120276baa7874d5aa0351a053598ab0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.143311 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb2420c2ca4eefa11a882651d7f9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.146728 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 901d22dfa4fd4cf2b574875ef4f8135b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.150758 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120276baa7874d5aa0351a053598ab0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:37.156878 12802 tablet_bootstrap.cc:492] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:37.158758 12802 tablet_bootstrap.cc:654] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.165313 12802 tablet_bootstrap.cc:492] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:37.165551 12802 ts_tablet_manager.cc:1367] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 14:01:37.166375 12802 raft_consensus.cc:348] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.166615 12802 raft_consensus.cc:374] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.166713 12802 raft_consensus.cc:729] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.166983 12802 consensus_queue.cc:260] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.168452 12802 ts_tablet_manager.cc:1398] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:37.172708 12802 tablet_bootstrap.cc:492] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:37.174475 12802 tablet_bootstrap.cc:654] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.176993 12804 tablet_bootstrap.cc:492] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:37.177999 12802 tablet_bootstrap.cc:492] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:37.178262 12802 ts_tablet_manager.cc:1367] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 14:01:37.178923 12804 tablet_bootstrap.cc:654] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.179270 12802 raft_consensus.cc:348] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.179585 12802 raft_consensus.cc:374] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.179704 12802 raft_consensus.cc:729] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.180068 12802 consensus_queue.cc:260] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.180318 12801 tablet_bootstrap.cc:492] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:37.182057 12802 ts_tablet_manager.cc:1398] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:01:37.182106 12801 tablet_bootstrap.cc:654] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.183449 12800 tablet_bootstrap.cc:492] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:37.184422 12804 tablet_bootstrap.cc:492] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:37.184590 12804 ts_tablet_manager.cc:1367] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:01:37.184741 12801 tablet_bootstrap.cc:492] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:37.184875 12801 ts_tablet_manager.cc:1367] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:01:37.185380 12804 raft_consensus.cc:348] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.185536 12804 raft_consensus.cc:374] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.185611 12804 raft_consensus.cc:729] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.185570 12801 raft_consensus.cc:348] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.185776 12801 raft_consensus.cc:374] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.185865 12801 raft_consensus.cc:729] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.185838 12804 consensus_queue.cc:260] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.186075 12801 consensus_queue.cc:260] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.186393 12804 ts_tablet_manager.cc:1398] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:37.186615 12801 ts_tablet_manager.cc:1398] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:37.186941 12804 tablet_bootstrap.cc:492] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:37.188362 12804 tablet_bootstrap.cc:654] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.189074 12800 tablet_bootstrap.cc:654] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.189621 12801 tablet_bootstrap.cc:492] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:37.191895 12801 tablet_bootstrap.cc:654] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.192226 12800 tablet_bootstrap.cc:492] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:37.192363 12800 ts_tablet_manager.cc:1367] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240501 14:01:37.193065 12800 raft_consensus.cc:348] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.193336 12800 raft_consensus.cc:374] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.193440 12800 raft_consensus.cc:729] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.193663 12800 consensus_queue.cc:260] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.193845 12804 tablet_bootstrap.cc:492] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:37.193843 12801 tablet_bootstrap.cc:492] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:37.194002 12804 ts_tablet_manager.cc:1367] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:37.194003 12801 ts_tablet_manager.cc:1367] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:37.194775 12800 ts_tablet_manager.cc:1398] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:37.194737 12804 raft_consensus.cc:348] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.194733 12801 raft_consensus.cc:348] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.194936 12804 raft_consensus.cc:374] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.194936 12801 raft_consensus.cc:374] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.195003 12804 raft_consensus.cc:729] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.195003 12801 raft_consensus.cc:729] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.195067 12800 tablet_bootstrap.cc:492] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:37.195261 12801 consensus_queue.cc:260] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.195261 12804 consensus_queue.cc:260] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.196771 12801 ts_tablet_manager.cc:1398] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:37.197007 12804 ts_tablet_manager.cc:1398] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:37.197163 12800 tablet_bootstrap.cc:654] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.197283 12804 tablet_bootstrap.cc:492] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:37.198545 12804 tablet_bootstrap.cc:654] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:37.199378 12800 tablet_bootstrap.cc:492] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:37.199532 12800 ts_tablet_manager.cc:1367] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:01:37.200101 12800 raft_consensus.cc:348] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.200294 12800 raft_consensus.cc:374] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.200376 12800 raft_consensus.cc:729] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.200618 12800 consensus_queue.cc:260] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.201278 12800 ts_tablet_manager.cc:1398] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:37.201514 12804 tablet_bootstrap.cc:492] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:37.201651 12804 ts_tablet_manager.cc:1367] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:37.202405 12804 raft_consensus.cc:348] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.202628 12804 raft_consensus.cc:374] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:37.202720 12804 raft_consensus.cc:729] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:37.202955 12804 consensus_queue.cc:260] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.204033 12804 ts_tablet_manager.cc:1398] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:37.315528 12470 raft_consensus.cc:482] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:37.315776 12470 raft_consensus.cc:504] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.316551 12470 leader_election.cc:288] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:37.317059 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:37.317065 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:37.317303 11735 raft_consensus.cc:2454] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:37.317297 11476 raft_consensus.cc:2454] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:37.317662 11803 leader_election.cc:302] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:37.318049 12470 raft_consensus.cc:2781] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:37.318154 12470 raft_consensus.cc:482] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:37.318257 12470 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.320302 12470 raft_consensus.cc:504] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.320883 12470 leader_election.cc:288] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:37.321276 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:37.321374 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:37.321496 11735 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.321557 11476 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.324275 11735 raft_consensus.cc:2454] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:37.324275 11476 raft_consensus.cc:2454] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:37.324707 11803 leader_election.cc:302] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:37.325006 12470 raft_consensus.cc:2781] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:37.325170 12470 raft_consensus.cc:686] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:37.325431 12470 consensus_queue.cc:237] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:37.328051 11326 catalog_manager.cc:5484] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e reported cstate change: term changed from 0 to 1, leader changed from <none> to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 1 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:37.370088 12727 raft_consensus.cc:482] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:37.370337 12727 raft_consensus.cc:504] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.372025 12727 leader_election.cc:288] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:37.380282 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:37.380281 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:37.380575 11735 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:37.380570 11607 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:37.381116 11409 leader_election.cc:302] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:37.381525 12727 raft_consensus.cc:2781] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:37.381645 12727 raft_consensus.cc:482] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:37.381726 12727 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.383728 12727 raft_consensus.cc:504] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.384349 12727 leader_election.cc:288] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:37.384816 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:37.384814 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:37.385011 11735 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.385017 11607 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.386864 11735 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:37.386863 11607 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:37.387284 11411 leader_election.cc:302] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:37.387589 12727 raft_consensus.cc:2781] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:37.387964 12727 raft_consensus.cc:686] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:37.388455 12727 consensus_queue.cc:237] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:37.391899 11326 catalog_manager.cc:5484] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:37.494324 12745 raft_consensus.cc:482] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:37.494565 12745 raft_consensus.cc:504] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.495151 12745 leader_election.cc:288] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:37.495728 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:37.495759 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:37.495957 11866 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:37.496013 11476 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:37.496413 11543 leader_election.cc:302] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:37.496690 12745 raft_consensus.cc:2781] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:37.496775 12745 raft_consensus.cc:482] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:37.496871 12745 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.499034 12745 raft_consensus.cc:504] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.499606 12745 leader_election.cc:288] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:37.500005 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:37.500051 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:37.500216 11866 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.500229 11476 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:37.501982 11866 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:37.501987 11476 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:37.502409 11543 leader_election.cc:302] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:37.502724 12745 raft_consensus.cc:2781] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:37.502892 12745 raft_consensus.cc:686] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:37.503170 12745 consensus_queue.cc:237] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:37.505853 11326 catalog_manager.cc:5484] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:37.710151 12470 consensus_queue.cc:1035] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.724365 12731 consensus_queue.cc:1035] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:37.858477 12820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:37.858855 12820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:37.866629 12820 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:01:37.939854 12727 consensus_queue.cc:1035] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.959944 12727 consensus_queue.cc:1035] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.976540 12745 consensus_queue.cc:1035] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:37.994385 12745 consensus_queue.cc:1035] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 358.169 ms
time per row: 0.179085 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:01:39.236219 11334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:39.245445 11847 tablet_service.cc:1467] Processing CreateTablet for tablet a69a0e9dd426490393039c70feb27b9d (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.245965 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69a0e9dd426490393039c70feb27b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.246677 11586 tablet_service.cc:1467] Processing CreateTablet for tablet a69a0e9dd426490393039c70feb27b9d (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.246824 11717 tablet_service.cc:1467] Processing CreateTablet for tablet a69a0e9dd426490393039c70feb27b9d (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.246950 11715 tablet_service.cc:1467] Processing CreateTablet for tablet a08f72693051415bb8956a9dd7f43476 (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.247109 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69a0e9dd426490393039c70feb27b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.247236 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69a0e9dd426490393039c70feb27b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.247437 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08f72693051415bb8956a9dd7f43476. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.251897 11454 tablet_service.cc:1467] Processing CreateTablet for tablet 47ce26abfa4142d4a83dbe8705f93b41 (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.252158 11585 tablet_service.cc:1467] Processing CreateTablet for tablet a08f72693051415bb8956a9dd7f43476 (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.252339 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ce26abfa4142d4a83dbe8705f93b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.252535 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08f72693051415bb8956a9dd7f43476. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.257369 12890 tablet_bootstrap.cc:492] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:39.257853 11845 tablet_service.cc:1467] Processing CreateTablet for tablet a08f72693051415bb8956a9dd7f43476 (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.258285 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08f72693051415bb8956a9dd7f43476. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.258818 11587 tablet_service.cc:1467] Processing CreateTablet for tablet 47ce26abfa4142d4a83dbe8705f93b41 (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.259253 11587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ce26abfa4142d4a83dbe8705f93b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.259258 12890 tablet_bootstrap.cc:654] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.262665 11846 tablet_service.cc:1467] Processing CreateTablet for tablet 47ce26abfa4142d4a83dbe8705f93b41 (DEFAULT_TABLE table=rebalance_test_table_5 [id=74a7273b4ca947eda3a0a63ffe1695ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:39.263168 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ce26abfa4142d4a83dbe8705f93b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:39.263345 12889 tablet_bootstrap.cc:492] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:39.265020 12889 tablet_bootstrap.cc:654] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.267395 12890 tablet_bootstrap.cc:492] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:39.267639 12890 ts_tablet_manager.cc:1367] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240501 14:01:39.269016 12890 raft_consensus.cc:348] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.269302 12890 raft_consensus.cc:374] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.269538 12890 raft_consensus.cc:729] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.270988 12890 consensus_queue.cc:260] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.272651 12890 ts_tablet_manager.cc:1398] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:01:39.278247 12890 tablet_bootstrap.cc:492] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:39.282075 12890 tablet_bootstrap.cc:654] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.283819 12894 tablet_bootstrap.cc:492] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:39.285766 12889 tablet_bootstrap.cc:492] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:39.286043 12889 ts_tablet_manager.cc:1367] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.007s
I20240501 14:01:39.286262 12894 tablet_bootstrap.cc:654] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.286798 12891 tablet_bootstrap.cc:492] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:39.287254 12889 raft_consensus.cc:348] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.287483 12889 raft_consensus.cc:374] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.287606 12889 raft_consensus.cc:729] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.287931 12889 consensus_queue.cc:260] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.288211 12890 tablet_bootstrap.cc:492] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:39.288393 12891 tablet_bootstrap.cc:654] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.288425 12890 ts_tablet_manager.cc:1367] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 14:01:39.289222 12890 raft_consensus.cc:348] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.289539 12890 raft_consensus.cc:374] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.289655 12890 raft_consensus.cc:729] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.290032 12890 consensus_queue.cc:260] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.291529 12890 ts_tablet_manager.cc:1398] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:39.291863 12890 tablet_bootstrap.cc:492] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:39.292161 12889 ts_tablet_manager.cc:1398] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:39.295254 12890 tablet_bootstrap.cc:654] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.296177 12889 tablet_bootstrap.cc:492] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:39.297997 12889 tablet_bootstrap.cc:654] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.299311 12891 tablet_bootstrap.cc:492] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:39.299957 12891 ts_tablet_manager.cc:1367] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240501 14:01:39.300879 12891 raft_consensus.cc:348] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.301224 12891 raft_consensus.cc:374] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.301359 12891 raft_consensus.cc:729] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.301695 12891 consensus_queue.cc:260] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.303400 12891 ts_tablet_manager.cc:1398] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:39.304711 12890 tablet_bootstrap.cc:492] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:39.304742 12889 tablet_bootstrap.cc:492] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:39.304754 12894 tablet_bootstrap.cc:492] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:39.304890 12889 ts_tablet_manager.cc:1367] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 14:01:39.304939 12890 ts_tablet_manager.cc:1367] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240501 14:01:39.304944 12894 ts_tablet_manager.cc:1367] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.017s
I20240501 14:01:39.305681 12889 raft_consensus.cc:348] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.305708 12890 raft_consensus.cc:348] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.305709 12894 raft_consensus.cc:348] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.305892 12890 raft_consensus.cc:374] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.305902 12894 raft_consensus.cc:374] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.305991 12890 raft_consensus.cc:729] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.306041 12894 raft_consensus.cc:729] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.306044 12889 raft_consensus.cc:374] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.306170 12889 raft_consensus.cc:729] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.306272 12890 consensus_queue.cc:260] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.306321 12894 consensus_queue.cc:260] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.306924 12890 ts_tablet_manager.cc:1398] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:39.307054 12894 ts_tablet_manager.cc:1398] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:39.306458 12889 consensus_queue.cc:260] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.307385 12894 tablet_bootstrap.cc:492] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:39.309054 12894 tablet_bootstrap.cc:654] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.309460 12889 ts_tablet_manager.cc:1398] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:01:39.311738 12894 tablet_bootstrap.cc:492] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:39.311914 12894 ts_tablet_manager.cc:1367] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:01:39.312672 12894 raft_consensus.cc:348] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.312889 12894 raft_consensus.cc:374] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.312976 12894 raft_consensus.cc:729] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.313244 12894 consensus_queue.cc:260] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.313797 12894 ts_tablet_manager.cc:1398] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:39.314069 12894 tablet_bootstrap.cc:492] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:39.315510 12894 tablet_bootstrap.cc:654] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:39.317178 12894 tablet_bootstrap.cc:492] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:39.317327 12894 ts_tablet_manager.cc:1367] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:39.317894 12727 raft_consensus.cc:482] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:39.317916 12894 raft_consensus.cc:348] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.318089 12894 raft_consensus.cc:374] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:39.318177 12894 raft_consensus.cc:729] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:39.318069 12727 raft_consensus.cc:504] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.318378 12894 consensus_queue.cc:260] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.318861 12894 ts_tablet_manager.cc:1398] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:01:39.318868 12727 leader_election.cc:288] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:39.320102 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:39.320391 11607 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:39.320992 11411 leader_election.cc:302] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:39.321326 12727 raft_consensus.cc:2781] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:39.321439 12727 raft_consensus.cc:482] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:39.321530 12727 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.323964 12727 raft_consensus.cc:504] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.324649 12727 leader_election.cc:288] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:39.325191 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:39.325443 11607 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.327751 11607 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:39.328188 11411 leader_election.cc:302] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:39.328503 12727 raft_consensus.cc:2781] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:39.329066 12727 raft_consensus.cc:686] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:39.329547 12727 consensus_queue.cc:237] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:39.332906 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:39.332703 11331 catalog_manager.cc:5484] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:39.333045 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:39.333194 11867 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:39.402987 12745 raft_consensus.cc:482] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:39.403196 12745 raft_consensus.cc:504] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.403882 12745 leader_election.cc:288] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:39.404443 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:39.404454 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:39.404652 11737 raft_consensus.cc:2454] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:39.404670 11866 raft_consensus.cc:2454] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:39.405076 11543 leader_election.cc:302] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:39.405369 12745 raft_consensus.cc:2781] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:39.405473 12745 raft_consensus.cc:482] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:39.405560 12745 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.407457 12745 raft_consensus.cc:504] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.408026 12745 leader_election.cc:288] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:39.408602 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:39.408586 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:39.408797 11737 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.408813 11866 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.411628 11866 raft_consensus.cc:2454] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:39.411640 11737 raft_consensus.cc:2454] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:39.412130 11543 leader_election.cc:302] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:39.412442 12745 raft_consensus.cc:2781] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:39.412806 12745 raft_consensus.cc:686] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:39.413122 12745 consensus_queue.cc:237] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.416097 11331 catalog_manager.cc:5484] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:39.536763 12621 raft_consensus.cc:482] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:39.536958 12621 raft_consensus.cc:504] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.537590 12621 leader_election.cc:288] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:39.538216 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:39.538194 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:39.538437 11866 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:39.538458 11607 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:39.538913 11673 leader_election.cc:302] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:39.539198 12621 raft_consensus.cc:2781] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:39.539312 12621 raft_consensus.cc:482] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:39.539394 12621 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.541242 12621 raft_consensus.cc:504] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.541821 12621 leader_election.cc:288] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:39.542376 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:39.542357 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:39.542596 11866 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.542622 11607 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.544968 11866 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:39.545357 11673 leader_election.cc:302] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:39.545627 12621 raft_consensus.cc:2781] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:39.545961 12621 raft_consensus.cc:686] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:39.546265 12621 consensus_queue.cc:237] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:39.549451 11331 catalog_manager.cc:5484] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 1 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:39.572588 11607 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:39.808609 11866 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:39.811481 12727 consensus_queue.cc:1035] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:39.821403 12904 consensus_queue.cc:1035] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:39.905764 12910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:39.906102 12910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:39.913530 12910 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:39.932322 12621 consensus_queue.cc:1035] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:39.938150 12621 consensus_queue.cc:1035] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240501 14:01:39.979816 12745 consensus_queue.cc:1035] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:40.001603 12745 consensus_queue.cc:1035] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 264.685 ms
time per row: 0.132342 ms
I20240501 14:01:40.328595 11332 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:40.337432 11586 tablet_service.cc:1467] Processing CreateTablet for tablet 08f0000ac0254bf3be818261a9a1a585 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.337931 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f0000ac0254bf3be818261a9a1a585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.337925 11717 tablet_service.cc:1467] Processing CreateTablet for tablet 08f0000ac0254bf3be818261a9a1a585 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.338295 11715 tablet_service.cc:1467] Processing CreateTablet for tablet cecca4823b0a4f1abdfdc3941f5942ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.338752 11454 tablet_service.cc:1467] Processing CreateTablet for tablet 08f0000ac0254bf3be818261a9a1a585 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.339151 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f0000ac0254bf3be818261a9a1a585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.339510 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 5b316e036f734208a0cfab91895239b1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.340127 11456 tablet_service.cc:1467] Processing CreateTablet for tablet 5b316e036f734208a0cfab91895239b1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.338395 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f0000ac0254bf3be818261a9a1a585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.340514 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b316e036f734208a0cfab91895239b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.342871 11455 tablet_service.cc:1467] Processing CreateTablet for tablet cecca4823b0a4f1abdfdc3941f5942ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.343225 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cecca4823b0a4f1abdfdc3941f5942ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.343197 11845 tablet_service.cc:1467] Processing CreateTablet for tablet cecca4823b0a4f1abdfdc3941f5942ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.343305 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cecca4823b0a4f1abdfdc3941f5942ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.343585 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cecca4823b0a4f1abdfdc3941f5942ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.346674 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b316e036f734208a0cfab91895239b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.353998 12980 tablet_bootstrap.cc:492] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:40.355628 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 5b316e036f734208a0cfab91895239b1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f354e8094bd14e5cb393e8b96f72e5e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:40.355830 12980 tablet_bootstrap.cc:654] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.356079 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b316e036f734208a0cfab91895239b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:40.357477 12977 tablet_bootstrap.cc:492] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:40.357611 12978 tablet_bootstrap.cc:492] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:40.359223 12977 tablet_bootstrap.cc:654] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.359799 12978 tablet_bootstrap.cc:654] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.361567 12980 tablet_bootstrap.cc:492] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:40.361739 12980 ts_tablet_manager.cc:1367] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:01:40.361971 12979 tablet_bootstrap.cc:492] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:40.362614 12980 raft_consensus.cc:348] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.362951 12980 raft_consensus.cc:374] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.363109 12980 raft_consensus.cc:729] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.363442 12980 consensus_queue.cc:260] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.364187 12979 tablet_bootstrap.cc:654] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.364842 12980 ts_tablet_manager.cc:1398] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:40.368110 12978 tablet_bootstrap.cc:492] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:40.368283 12978 ts_tablet_manager.cc:1367] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 14:01:40.369150 12978 raft_consensus.cc:348] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.369334 12978 raft_consensus.cc:374] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.369424 12978 raft_consensus.cc:729] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.369699 12978 consensus_queue.cc:260] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.370808 12978 ts_tablet_manager.cc:1398] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:01:40.371115 12978 tablet_bootstrap.cc:492] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:40.373857 12980 tablet_bootstrap.cc:492] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:40.375070 12978 tablet_bootstrap.cc:654] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.375504 12980 tablet_bootstrap.cc:654] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.378077 12977 tablet_bootstrap.cc:492] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:40.378314 12977 ts_tablet_manager.cc:1367] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.011s
I20240501 14:01:40.379091 12978 tablet_bootstrap.cc:492] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:40.379194 12979 tablet_bootstrap.cc:492] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:40.379257 12978 ts_tablet_manager.cc:1367] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240501 14:01:40.379209 12977 raft_consensus.cc:348] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.379357 12979 ts_tablet_manager.cc:1367] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240501 14:01:40.379420 12977 raft_consensus.cc:374] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.379514 12977 raft_consensus.cc:729] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.379750 12977 consensus_queue.cc:260] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.380059 12979 raft_consensus.cc:348] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.380084 12978 raft_consensus.cc:348] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.380295 12978 raft_consensus.cc:374] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.380301 12979 raft_consensus.cc:374] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.380393 12979 raft_consensus.cc:729] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.380410 12978 raft_consensus.cc:729] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.380642 12979 consensus_queue.cc:260] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.380690 12978 consensus_queue.cc:260] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.381258 12978 ts_tablet_manager.cc:1398] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:40.381567 12978 tablet_bootstrap.cc:492] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:40.381759 12980 tablet_bootstrap.cc:492] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:40.381948 12980 ts_tablet_manager.cc:1367] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:01:40.382776 12980 raft_consensus.cc:348] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.382933 12979 ts_tablet_manager.cc:1398] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:40.383067 12980 raft_consensus.cc:374] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.383204 12980 raft_consensus.cc:729] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.383226 12978 tablet_bootstrap.cc:654] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.383502 12980 consensus_queue.cc:260] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.385061 12980 ts_tablet_manager.cc:1398] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:40.385686 12977 ts_tablet_manager.cc:1398] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:01:40.387725 12977 tablet_bootstrap.cc:492] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:40.388532 12978 tablet_bootstrap.cc:492] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:40.388733 12978 ts_tablet_manager.cc:1367] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:01:40.389575 12977 tablet_bootstrap.cc:654] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.389732 12978 raft_consensus.cc:348] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.389950 12978 raft_consensus.cc:374] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.390058 12978 raft_consensus.cc:729] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.390269 12978 consensus_queue.cc:260] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.391315 12978 ts_tablet_manager.cc:1398] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:40.391530 12977 tablet_bootstrap.cc:492] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:40.391701 12977 ts_tablet_manager.cc:1367] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:40.392547 12977 raft_consensus.cc:348] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.392766 12977 raft_consensus.cc:374] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.392846 12977 raft_consensus.cc:729] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.393077 12977 consensus_queue.cc:260] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.393632 12977 ts_tablet_manager.cc:1398] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:40.394038 12977 tablet_bootstrap.cc:492] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:40.395624 12977 tablet_bootstrap.cc:654] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:40.397429 12977 tablet_bootstrap.cc:492] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:40.397547 12977 ts_tablet_manager.cc:1367] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:40.398228 12977 raft_consensus.cc:348] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.398394 12977 raft_consensus.cc:374] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:40.398478 12977 raft_consensus.cc:729] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:40.398707 12977 consensus_queue.cc:260] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.399186 12977 ts_tablet_manager.cc:1398] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:40.423003 12904 raft_consensus.cc:482] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:40.423198 12904 raft_consensus.cc:504] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.423779 12904 leader_election.cc:288] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:40.424260 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:40.424357 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:40.424515 11737 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:40.424592 11606 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:40.424953 11409 leader_election.cc:302] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:40.425246 12904 raft_consensus.cc:2781] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:40.425336 12904 raft_consensus.cc:482] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:40.425410 12904 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.427273 12904 raft_consensus.cc:504] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.427832 12904 leader_election.cc:288] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:40.428318 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:40.428311 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:40.428536 11737 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.428536 11606 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.431349 11606 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:40.431349 11737 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:40.431851 11411 leader_election.cc:302] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:40.432247 12904 raft_consensus.cc:2781] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:40.432487 12904 raft_consensus.cc:686] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:40.432801 12904 consensus_queue.cc:237] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:40.436239 11331 catalog_manager.cc:5484] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:40.474839 12904 raft_consensus.cc:482] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:40.475036 12904 raft_consensus.cc:504] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.475791 12904 leader_election.cc:288] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:40.476280 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:40.476322 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:40.476563 11606 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:40.476567 11737 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:40.477010 11409 leader_election.cc:302] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:40.477301 12904 raft_consensus.cc:2781] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:40.477411 12904 raft_consensus.cc:482] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:40.477500 12904 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.479394 12904 raft_consensus.cc:504] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.479969 12904 leader_election.cc:288] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:40.480441 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:40.480504 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:40.480684 11737 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.480715 11606 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.483484 11606 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:40.483945 11411 leader_election.cc:302] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:40.484248 11737 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:40.484458 12727 raft_consensus.cc:2781] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:40.484619 12727 raft_consensus.cc:686] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:40.484872 12727 consensus_queue.cc:237] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:40.487958 11331 catalog_manager.cc:5484] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:40.492511 12904 raft_consensus.cc:482] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:40.492687 12904 raft_consensus.cc:504] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.493580 12904 leader_election.cc:288] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:40.493825 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:40.493875 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:40.494078 11735 raft_consensus.cc:2454] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:40.494073 11865 raft_consensus.cc:2454] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:40.494575 11412 leader_election.cc:302] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:40.494868 12904 raft_consensus.cc:2781] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:40.495105 12904 raft_consensus.cc:482] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:40.495208 12904 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.496999 12904 raft_consensus.cc:504] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.497596 12904 leader_election.cc:288] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:40.498034 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:40.498056 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:40.498222 11865 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.498241 11735 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:40.500485 11865 raft_consensus.cc:2454] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:40.500741 11735 raft_consensus.cc:2454] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:40.500926 11412 leader_election.cc:302] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:40.501238 12904 raft_consensus.cc:2781] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:40.501371 12904 raft_consensus.cc:686] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:40.501653 12904 consensus_queue.cc:237] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:40.504463 11331 catalog_manager.cc:5484] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:40.851958 12904 consensus_queue.cc:1035] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:40.859994 12904 consensus_queue.cc:1035] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:40.871222 12993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:40.871593 12993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:40.879256 12993 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:01:40.908440 12904 consensus_queue.cc:1035] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:40.913242 12914 consensus_queue.cc:1035] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240501 14:01:41.003201 13010 consensus_queue.cc:1035] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:41.016834 13010 consensus_queue.cc:1035] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.796 ms
time per row: 0.117898 ms
I20240501 14:01:41.286199 11333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:41.295478 11845 tablet_service.cc:1467] Processing CreateTablet for tablet 537e656ba83e47d3b990e0b7a7e8e9e2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.295526 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 537e656ba83e47d3b990e0b7a7e8e9e2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.295966 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537e656ba83e47d3b990e0b7a7e8e9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.295975 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537e656ba83e47d3b990e0b7a7e8e9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.297086 11715 tablet_service.cc:1467] Processing CreateTablet for tablet 5062be5a545c430b8f9277e77c51e11e (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.297502 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5062be5a545c430b8f9277e77c51e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.298954 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 537e656ba83e47d3b990e0b7a7e8e9e2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.299389 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537e656ba83e47d3b990e0b7a7e8e9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.302836 11586 tablet_service.cc:1467] Processing CreateTablet for tablet 5062be5a545c430b8f9277e77c51e11e (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.303304 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5062be5a545c430b8f9277e77c51e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.303810 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 5062be5a545c430b8f9277e77c51e11e (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.304241 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5062be5a545c430b8f9277e77c51e11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.298458 11716 tablet_service.cc:1467] Processing CreateTablet for tablet c6c91578d215423c88f8759871a591ca (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.307473 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6c91578d215423c88f8759871a591ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.310899 11456 tablet_service.cc:1467] Processing CreateTablet for tablet c6c91578d215423c88f8759871a591ca (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.311337 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6c91578d215423c88f8759871a591ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.314872 11846 tablet_service.cc:1467] Processing CreateTablet for tablet c6c91578d215423c88f8759871a591ca (DEFAULT_TABLE table=rebalance_test_table_7 [id=80d6994ea4e4494f87050eb9b4cdb131]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:41.315387 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6c91578d215423c88f8759871a591ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:41.317638 13055 tablet_bootstrap.cc:492] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:41.319522 13055 tablet_bootstrap.cc:654] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.320086 13058 tablet_bootstrap.cc:492] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:41.321830 13057 tablet_bootstrap.cc:492] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:41.321884 13058 tablet_bootstrap.cc:654] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.324839 13057 tablet_bootstrap.cc:654] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.333424 13056 tablet_bootstrap.cc:492] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:41.334617 13058 tablet_bootstrap.cc:492] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:41.334834 13058 ts_tablet_manager.cc:1367] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240501 14:01:41.335590 13056 tablet_bootstrap.cc:654] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.335669 13058 raft_consensus.cc:348] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.335889 13058 raft_consensus.cc:374] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.335983 13058 raft_consensus.cc:729] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.336236 13058 consensus_queue.cc:260] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.336438 13057 tablet_bootstrap.cc:492] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:41.336611 13057 ts_tablet_manager.cc:1367] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240501 14:01:41.336810 13058 ts_tablet_manager.cc:1398] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:41.337503 13057 raft_consensus.cc:348] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.337769 13057 raft_consensus.cc:374] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.337905 13057 raft_consensus.cc:729] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.338157 13055 tablet_bootstrap.cc:492] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:41.338210 13057 consensus_queue.cc:260] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.338383 13055 ts_tablet_manager.cc:1367] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.003s
I20240501 14:01:41.339531 13055 raft_consensus.cc:348] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.339849 13055 raft_consensus.cc:374] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.339975 13055 raft_consensus.cc:729] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.340444 13055 consensus_queue.cc:260] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.341229 13055 ts_tablet_manager.cc:1398] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:41.341591 13055 tablet_bootstrap.cc:492] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:41.342563 13058 tablet_bootstrap.cc:492] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:41.343354 13055 tablet_bootstrap.cc:654] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.344208 13058 tablet_bootstrap.cc:654] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.344204 13057 ts_tablet_manager.cc:1398] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:01:41.346792 13057 tablet_bootstrap.cc:492] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:41.348378 13057 tablet_bootstrap.cc:654] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.348706 13056 tablet_bootstrap.cc:492] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:41.348923 13056 ts_tablet_manager.cc:1367] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20240501 14:01:41.350178 13058 tablet_bootstrap.cc:492] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:41.349815 13056 raft_consensus.cc:348] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.350353 13058 ts_tablet_manager.cc:1367] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20240501 14:01:41.350365 13056 raft_consensus.cc:374] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.350492 13056 raft_consensus.cc:729] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.350852 13056 consensus_queue.cc:260] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.351222 13058 raft_consensus.cc:348] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.351456 13058 raft_consensus.cc:374] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.351536 13058 raft_consensus.cc:729] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.351568 13055 tablet_bootstrap.cc:492] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:41.351716 13055 ts_tablet_manager.cc:1367] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 14:01:41.351806 13058 consensus_queue.cc:260] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.352537 13058 ts_tablet_manager.cc:1398] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:41.352464 13055 raft_consensus.cc:348] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.352734 13055 raft_consensus.cc:374] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.352849 13055 raft_consensus.cc:729] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.353137 13055 consensus_queue.cc:260] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.353761 13057 tablet_bootstrap.cc:492] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:41.353798 13055 ts_tablet_manager.cc:1398] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:41.353938 13057 ts_tablet_manager.cc:1367] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:01:41.354588 13057 raft_consensus.cc:348] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.354763 13057 raft_consensus.cc:374] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.354835 13057 raft_consensus.cc:729] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.355115 13057 consensus_queue.cc:260] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.355733 13057 ts_tablet_manager.cc:1398] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:41.358692 13056 ts_tablet_manager.cc:1398] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:01:41.359010 13056 tablet_bootstrap.cc:492] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:41.360656 13056 tablet_bootstrap.cc:654] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.363092 13056 tablet_bootstrap.cc:492] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:41.363260 13056 ts_tablet_manager.cc:1367] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:41.364017 13056 raft_consensus.cc:348] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.364223 13056 raft_consensus.cc:374] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.364296 13056 raft_consensus.cc:729] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.364531 13056 consensus_queue.cc:260] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.365094 13056 ts_tablet_manager.cc:1398] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:01:41.365434 13056 tablet_bootstrap.cc:492] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:41.366964 13056 tablet_bootstrap.cc:654] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:41.368757 13056 tablet_bootstrap.cc:492] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:41.368888 13056 ts_tablet_manager.cc:1367] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:41.369491 13056 raft_consensus.cc:348] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.369683 13056 raft_consensus.cc:374] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:41.369771 13056 raft_consensus.cc:729] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:41.369977 13056 consensus_queue.cc:260] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.370393 13056 ts_tablet_manager.cc:1398] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:41.402278 12929 raft_consensus.cc:482] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:41.402462 12929 raft_consensus.cc:504] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.403154 12929 leader_election.cc:288] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:41.403650 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:41.403661 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:41.403851 11607 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:41.403934 11474 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:41.404269 11802 leader_election.cc:302] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:41.404544 12929 raft_consensus.cc:2781] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:41.404647 12929 raft_consensus.cc:482] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:41.404752 12929 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.406697 12929 raft_consensus.cc:504] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.407356 12929 leader_election.cc:288] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:41.407840 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:41.407989 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:41.408061 11474 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.408195 11607 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.411010 11474 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:41.411010 11607 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:41.411471 11802 leader_election.cc:302] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:41.411820 12929 raft_consensus.cc:2781] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:41.412256 12929 raft_consensus.cc:686] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:41.412585 12929 consensus_queue.cc:237] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.415493 11332 catalog_manager.cc:5484] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 0 to 1, leader changed from <none> to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 1 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:41.484778 12621 raft_consensus.cc:482] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:41.485013 12621 raft_consensus.cc:504] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.485774 12621 leader_election.cc:288] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:41.486346 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:41.486379 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:41.486630 11607 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:41.486646 11865 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:41.487082 11673 leader_election.cc:302] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:41.487396 12621 raft_consensus.cc:2781] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:41.487505 12621 raft_consensus.cc:482] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:41.487617 12621 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.490589 12621 raft_consensus.cc:504] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.491439 12621 leader_election.cc:288] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:41.491876 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:41.492111 11865 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.492177 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:41.492391 11607 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.495039 11865 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:41.495039 11607 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:41.495481 11672 leader_election.cc:302] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:41.495791 12621 raft_consensus.cc:2781] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:41.496297 12621 raft_consensus.cc:686] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:41.496608 12621 consensus_queue.cc:237] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:41.500262 11332 catalog_manager.cc:5484] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 1 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:41.511873 12914 raft_consensus.cc:482] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:41.512135 12914 raft_consensus.cc:504] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.512923 12914 leader_election.cc:288] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:41.513449 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:41.513468 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:41.513684 11865 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:41.513706 11737 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:41.514242 11412 leader_election.cc:302] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:41.514598 12914 raft_consensus.cc:2781] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:41.514695 12914 raft_consensus.cc:482] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:41.514801 12914 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.516803 12914 raft_consensus.cc:504] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.517357 12914 leader_election.cc:288] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:41.517799 11865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:41.517832 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:41.517982 11865 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.518026 11737 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:41.519924 11737 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:41.519937 11865 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:41.520372 11409 leader_election.cc:302] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:41.520664 12914 raft_consensus.cc:2781] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:41.520812 12914 raft_consensus.cc:686] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:41.521086 12914 consensus_queue.cc:237] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:41.524080 11332 catalog_manager.cc:5484] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:41.902283 12621 consensus_queue.cc:1035] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:41.904742 13073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:41.905187 13073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:01:41.908282 12621 consensus_queue.cc:1035] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:41.916697 13073 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:01:41.971547 12929 consensus_queue.cc:1035] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:41.974686 12929 consensus_queue.cc:1035] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:41.983906 12914 consensus_queue.cc:1035] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:42.024917 13043 consensus_queue.cc:1035] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 236.61 ms
time per row: 0.118305 ms
I20240501 14:01:42.297736 11332 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:42.306598 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 658947a6b41844c593994a56bf20d0b0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.306669 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 658947a6b41844c593994a56bf20d0b0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.307101 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658947a6b41844c593994a56bf20d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.307121 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658947a6b41844c593994a56bf20d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.307264 11845 tablet_service.cc:1467] Processing CreateTablet for tablet 658947a6b41844c593994a56bf20d0b0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.307619 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 658947a6b41844c593994a56bf20d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.308359 11846 tablet_service.cc:1467] Processing CreateTablet for tablet b596b92c95d24211973cd278eb57c5ca (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.308714 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b596b92c95d24211973cd278eb57c5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.311360 11715 tablet_service.cc:1467] Processing CreateTablet for tablet b596b92c95d24211973cd278eb57c5ca (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.311755 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b596b92c95d24211973cd278eb57c5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.314970 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 63a9c838c4304ff0823e9cffff09bc58 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.315393 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a9c838c4304ff0823e9cffff09bc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.319301 11586 tablet_service.cc:1467] Processing CreateTablet for tablet 63a9c838c4304ff0823e9cffff09bc58 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.319473 11456 tablet_service.cc:1467] Processing CreateTablet for tablet 63a9c838c4304ff0823e9cffff09bc58 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.319766 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a9c838c4304ff0823e9cffff09bc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.319905 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a9c838c4304ff0823e9cffff09bc58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.319489 11454 tablet_service.cc:1467] Processing CreateTablet for tablet b596b92c95d24211973cd278eb57c5ca (DEFAULT_TABLE table=rebalance_test_table_8 [id=fde4d834efeb440d8d2f82e212e426d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:42.327711 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b596b92c95d24211973cd278eb57c5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:42.328891 13137 tablet_bootstrap.cc:492] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:42.331413 13137 tablet_bootstrap.cc:654] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.336692 13138 tablet_bootstrap.cc:492] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:42.337044 13140 tablet_bootstrap.cc:492] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:42.337569 13139 tablet_bootstrap.cc:492] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:42.338665 13138 tablet_bootstrap.cc:654] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.339118 13139 tablet_bootstrap.cc:654] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.340349 13137 tablet_bootstrap.cc:492] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:42.340570 13137 ts_tablet_manager.cc:1367] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 14:01:42.341223 13140 tablet_bootstrap.cc:654] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.343505 13137 raft_consensus.cc:348] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.343887 13137 raft_consensus.cc:374] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.344023 13137 raft_consensus.cc:729] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.344413 13137 consensus_queue.cc:260] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.347792 13140 tablet_bootstrap.cc:492] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:42.348052 13140 ts_tablet_manager.cc:1367] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 14:01:42.349100 13137 ts_tablet_manager.cc:1398] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:42.349004 13140 raft_consensus.cc:348] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.349236 13140 raft_consensus.cc:374] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.349328 13140 raft_consensus.cc:729] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.351419 13140 consensus_queue.cc:260] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.352646 13137 tablet_bootstrap.cc:492] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:42.354593 13137 tablet_bootstrap.cc:654] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.356832 13140 ts_tablet_manager.cc:1398] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:01:42.357403 13138 tablet_bootstrap.cc:492] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:42.357753 13138 ts_tablet_manager.cc:1367] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.012s
I20240501 14:01:42.359257 13138 raft_consensus.cc:348] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.359691 13138 raft_consensus.cc:374] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.359838 13138 raft_consensus.cc:729] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.360213 13138 consensus_queue.cc:260] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.360456 13137 tablet_bootstrap.cc:492] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:42.360713 13140 tablet_bootstrap.cc:492] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:42.361171 13138 ts_tablet_manager.cc:1398] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:42.360631 13137 ts_tablet_manager.cc:1367] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 14:01:42.362175 13137 raft_consensus.cc:348] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.362449 13137 raft_consensus.cc:374] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.362545 13137 raft_consensus.cc:729] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.362834 13137 consensus_queue.cc:260] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.362993 13138 tablet_bootstrap.cc:492] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:42.363817 13139 tablet_bootstrap.cc:492] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:42.364313 13140 tablet_bootstrap.cc:654] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.364606 13138 tablet_bootstrap.cc:654] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.365211 13139 ts_tablet_manager.cc:1367] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.010s
I20240501 14:01:42.365546 13137 ts_tablet_manager.cc:1398] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:42.365856 13137 tablet_bootstrap.cc:492] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:42.366194 13139 raft_consensus.cc:348] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.366417 13139 raft_consensus.cc:374] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.366523 13139 raft_consensus.cc:729] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.366776 13139 consensus_queue.cc:260] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.367409 13139 ts_tablet_manager.cc:1398] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:42.367727 13139 tablet_bootstrap.cc:492] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:42.368048 13137 tablet_bootstrap.cc:654] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.369418 13139 tablet_bootstrap.cc:654] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:42.371610 13140 tablet_bootstrap.cc:492] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:42.371798 13140 ts_tablet_manager.cc:1367] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240501 14:01:42.372622 13140 raft_consensus.cc:348] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.372866 13140 raft_consensus.cc:374] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.372951 13140 raft_consensus.cc:729] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.373273 13140 consensus_queue.cc:260] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.373963 13140 ts_tablet_manager.cc:1398] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:42.375784 13137 tablet_bootstrap.cc:492] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:42.375957 13137 ts_tablet_manager.cc:1367] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:01:42.376833 13137 raft_consensus.cc:348] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.377059 13137 raft_consensus.cc:374] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.377152 13137 raft_consensus.cc:729] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.377430 13137 consensus_queue.cc:260] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.378082 13137 ts_tablet_manager.cc:1398] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:42.379612 13139 tablet_bootstrap.cc:492] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:42.379666 13138 tablet_bootstrap.cc:492] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:42.379789 13139 ts_tablet_manager.cc:1367] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240501 14:01:42.379838 13138 ts_tablet_manager.cc:1367] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20240501 14:01:42.380590 13139 raft_consensus.cc:348] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.380514 13138 raft_consensus.cc:348] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.380806 13139 raft_consensus.cc:374] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.380872 13139 raft_consensus.cc:729] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.380892 13138 raft_consensus.cc:374] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:42.380983 13138 raft_consensus.cc:729] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:42.381116 13139 consensus_queue.cc:260] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.381278 13138 consensus_queue.cc:260] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.382565 13139 ts_tablet_manager.cc:1398] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 14:01:42.382885 13138 ts_tablet_manager.cc:1398] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:42.390316 12972 raft_consensus.cc:482] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:42.390480 12972 raft_consensus.cc:504] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.391121 12972 leader_election.cc:288] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:42.391620 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:42.391729 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:42.391845 11474 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:42.391997 11867 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:42.392247 11543 leader_election.cc:302] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:42.392551 12972 raft_consensus.cc:2781] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:42.392668 12972 raft_consensus.cc:482] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:42.392753 12972 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.394722 12972 raft_consensus.cc:504] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.395311 12972 leader_election.cc:288] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:42.395826 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:42.396076 11474 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.398052 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:42.398291 11867 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.398701 11474 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:42.399155 11543 leader_election.cc:302] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:42.399451 12972 raft_consensus.cc:2781] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:42.399605 12972 raft_consensus.cc:686] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:42.399878 12972 consensus_queue.cc:237] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:42.400499 11867 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:42.403182 11333 catalog_manager.cc:5484] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:42.408457 12914 raft_consensus.cc:482] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:42.408641 12914 raft_consensus.cc:504] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.409231 12914 leader_election.cc:288] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:42.409819 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:42.409792 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:42.410041 11867 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:42.410089 11735 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 0.
I20240501 14:01:42.410473 11412 leader_election.cc:302] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:42.410835 12914 raft_consensus.cc:2781] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:42.411121 12914 raft_consensus.cc:482] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:42.411242 12914 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.413553 12914 raft_consensus.cc:504] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.414108 12914 leader_election.cc:288] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:42.414561 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:42.414572 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:42.414753 11867 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.414759 11735 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.416688 11867 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:42.416673 11735 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 1.
I20240501 14:01:42.417105 11409 leader_election.cc:302] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:42.417388 12914 raft_consensus.cc:2781] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:42.417553 12914 raft_consensus.cc:686] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:42.417799 12914 consensus_queue.cc:237] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:42.420882 11333 catalog_manager.cc:5484] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 1 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:42.503656 12972 raft_consensus.cc:482] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:42.503861 12972 raft_consensus.cc:504] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.504455 12972 leader_election.cc:288] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:42.504920 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:42.504962 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:42.505126 11735 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:42.505149 11474 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:42.505594 11540 leader_election.cc:302] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:42.505883 12972 raft_consensus.cc:2781] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:42.505985 12972 raft_consensus.cc:482] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:42.506093 12972 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.508193 12972 raft_consensus.cc:504] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.508781 12972 leader_election.cc:288] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:42.509194 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:42.509342 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:42.509424 11735 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.509538 11474 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:42.512194 11474 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:42.512194 11735 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:42.512657 11540 leader_election.cc:302] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:42.512923 12972 raft_consensus.cc:2781] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:42.513103 12972 raft_consensus.cc:686] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:42.513350 12972 consensus_queue.cc:237] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:42.516170 11333 catalog_manager.cc:5484] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
W20240501 14:01:42.900774 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:42.901088 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:42.908489 13153 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:01:42.992282 12972 consensus_queue.cc:1035] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:42.994534 13133 consensus_queue.cc:1035] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:43.003383 12914 consensus_queue.cc:1035] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:43.007546 12914 consensus_queue.cc:1035] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:43.024659 13133 consensus_queue.cc:1035] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:43.074015 13187 consensus_queue.cc:1035] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 308.187 ms
time per row: 0.154093 ms
I20240501 14:01:43.367224 11326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32780:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:01:43.376758 11585 tablet_service.cc:1467] Processing CreateTablet for tablet 9df364f7a0374bdb9933d14a2b72c58e (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.376912 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 9df364f7a0374bdb9933d14a2b72c58e (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.377259 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df364f7a0374bdb9933d14a2b72c58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.377321 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df364f7a0374bdb9933d14a2b72c58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.378134 11715 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8df7567b084bb691ae69ba715e5d5d (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.378175 11587 tablet_service.cc:1467] Processing CreateTablet for tablet 7b44daf649544443944f1b54859ec381 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.378571 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8df7567b084bb691ae69ba715e5d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.378561 11587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b44daf649544443944f1b54859ec381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.382921 11586 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8df7567b084bb691ae69ba715e5d5d (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.383344 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8df7567b084bb691ae69ba715e5d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.385555 13219 tablet_bootstrap.cc:492] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:43.385715 11846 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8df7567b084bb691ae69ba715e5d5d (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.386200 11846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d8df7567b084bb691ae69ba715e5d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.387387 13219 tablet_bootstrap.cc:654] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.391001 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 7b44daf649544443944f1b54859ec381 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.391047 11845 tablet_service.cc:1467] Processing CreateTablet for tablet 9df364f7a0374bdb9933d14a2b72c58e (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.391484 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b44daf649544443944f1b54859ec381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.392102 11455 tablet_service.cc:1467] Processing CreateTablet for tablet 7b44daf649544443944f1b54859ec381 (DEFAULT_TABLE table=rebalance_test_table_9 [id=841f5ef6d0604763b1e47dce2a575e01]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:01:43.392503 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b44daf649544443944f1b54859ec381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.393039 13219 tablet_bootstrap.cc:492] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:43.393154 13219 ts_tablet_manager.cc:1367] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:01:43.393762 13219 raft_consensus.cc:348] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.393920 13219 raft_consensus.cc:374] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.393975 13219 raft_consensus.cc:729] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.394201 13219 consensus_queue.cc:260] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.394686 13219 ts_tablet_manager.cc:1398] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:01:43.395138 11845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df364f7a0374bdb9933d14a2b72c58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:43.398671 13222 tablet_bootstrap.cc:492] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8: Bootstrap starting.
I20240501 14:01:43.400446 13222 tablet_bootstrap.cc:654] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.405201 13220 tablet_bootstrap.cc:492] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:43.406921 13223 tablet_bootstrap.cc:492] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:43.407372 13220 tablet_bootstrap.cc:654] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.408954 13223 tablet_bootstrap.cc:654] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.408991 13222 tablet_bootstrap.cc:492] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8: No bootstrap required, opened a new log
I20240501 14:01:43.409227 13222 ts_tablet_manager.cc:1367] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 14:01:43.410092 13222 raft_consensus.cc:348] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.410398 13222 raft_consensus.cc:374] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.410560 13222 raft_consensus.cc:729] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.410868 13222 consensus_queue.cc:260] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.411517 13222 ts_tablet_manager.cc:1398] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:43.416968 13220 tablet_bootstrap.cc:492] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:43.417205 13223 tablet_bootstrap.cc:492] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:43.417225 13220 ts_tablet_manager.cc:1367] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240501 14:01:43.417430 13223 ts_tablet_manager.cc:1367] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240501 14:01:43.418155 13220 raft_consensus.cc:348] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.418640 13220 raft_consensus.cc:374] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.418803 13220 raft_consensus.cc:729] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.419121 13220 consensus_queue.cc:260] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.419227 13223 raft_consensus.cc:348] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.419486 13223 raft_consensus.cc:374] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.419682 13223 raft_consensus.cc:729] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.419953 13220 ts_tablet_manager.cc:1398] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:43.419963 13223 consensus_queue.cc:260] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.420418 13220 tablet_bootstrap.cc:492] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:43.420641 13223 ts_tablet_manager.cc:1398] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:01:43.422735 13220 tablet_bootstrap.cc:654] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.425264 13220 tablet_bootstrap.cc:492] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:43.425427 13220 ts_tablet_manager.cc:1367] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:01:43.426131 13220 raft_consensus.cc:348] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.426362 13220 raft_consensus.cc:374] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.426435 13220 raft_consensus.cc:729] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.426687 13220 consensus_queue.cc:260] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.427441 13220 ts_tablet_manager.cc:1398] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:43.428912 13223 tablet_bootstrap.cc:492] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:43.430400 13223 tablet_bootstrap.cc:654] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.431295 13220 tablet_bootstrap.cc:492] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6: Bootstrap starting.
I20240501 14:01:43.431459 13219 tablet_bootstrap.cc:492] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b: Bootstrap starting.
I20240501 14:01:43.432901 13220 tablet_bootstrap.cc:654] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.432916 13219 tablet_bootstrap.cc:654] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.434338 13223 tablet_bootstrap.cc:492] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:43.434561 13223 ts_tablet_manager.cc:1367] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:01:43.435104 13220 tablet_bootstrap.cc:492] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6: No bootstrap required, opened a new log
I20240501 14:01:43.435243 13220 ts_tablet_manager.cc:1367] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:01:43.435559 13219 tablet_bootstrap.cc:492] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b: No bootstrap required, opened a new log
I20240501 14:01:43.435554 13223 raft_consensus.cc:348] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.435716 13219 ts_tablet_manager.cc:1367] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:01:43.435788 13223 raft_consensus.cc:374] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.435869 13223 raft_consensus.cc:729] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.435907 13220 raft_consensus.cc:348] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.436115 13220 raft_consensus.cc:374] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.436196 13220 raft_consensus.cc:729] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.436121 13223 consensus_queue.cc:260] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.436417 13220 consensus_queue.cc:260] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.436722 13223 ts_tablet_manager.cc:1398] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:43.437601 13220 ts_tablet_manager.cc:1398] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:01:43.437528 13219 raft_consensus.cc:348] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.437729 13219 raft_consensus.cc:374] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.437795 13219 raft_consensus.cc:729] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.438011 13219 consensus_queue.cc:260] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.438665 13219 ts_tablet_manager.cc:1398] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:01:43.439190 13223 tablet_bootstrap.cc:492] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e: Bootstrap starting.
I20240501 14:01:43.440838 13223 tablet_bootstrap.cc:654] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:43.443675 13223 tablet_bootstrap.cc:492] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e: No bootstrap required, opened a new log
I20240501 14:01:43.443826 13223 ts_tablet_manager.cc:1367] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:01:43.444630 13223 raft_consensus.cc:348] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.444859 13223 raft_consensus.cc:374] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:43.444954 13223 raft_consensus.cc:729] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Initialized, Role: FOLLOWER
I20240501 14:01:43.445201 13223 consensus_queue.cc:260] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.445724 13223 ts_tablet_manager.cc:1398] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:01:43.445981 13214 raft_consensus.cc:482] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:43.446101 13214 raft_consensus.cc:504] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.446779 13214 leader_election.cc:288] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:43.447162 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:43.447170 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:43.447382 11867 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:43.447417 11606 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 0.
I20240501 14:01:43.447885 11672 leader_election.cc:302] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:43.448194 13214 raft_consensus.cc:2781] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:43.448324 13214 raft_consensus.cc:482] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:43.448405 13214 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.449370 13107 raft_consensus.cc:482] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:43.449502 13107 raft_consensus.cc:504] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.450073 13107 leader_election.cc:288] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:43.450493 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6" is_pre_election: true
I20240501 14:01:43.450606 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" is_pre_election: true
I20240501 14:01:43.450557 13214 raft_consensus.cc:504] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.450748 11607 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:43.450786 11737 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 0.
I20240501 14:01:43.451260 11800 leader_election.cc:302] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:43.451503 13107 raft_consensus.cc:2781] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:43.451608 13107 raft_consensus.cc:482] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:43.451701 13107 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.451936 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:43.452162 11867 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.454053 13107 raft_consensus.cc:504] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.454645 13107 leader_election.cc:288] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:43.454941 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:43.455124 11606 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.455233 11867 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:43.455857 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:43.455981 11673 leader_election.cc:302] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:43.456079 11735 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.456298 12621 raft_consensus.cc:2781] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:43.456296 13214 leader_election.cc:288] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 1 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:43.457573 11606 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 1.
I20240501 14:01:43.458683 11735 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:43.459141 11800 leader_election.cc:302] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:43.462742 12621 raft_consensus.cc:686] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:43.463210 13232 raft_consensus.cc:2781] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:43.463421 12621 consensus_queue.cc:237] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.463688 13232 raft_consensus.cc:686] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:43.464004 13232 consensus_queue.cc:237] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:43.465164 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:43.465387 11606 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.466804 11334 catalog_manager.cc:5484] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 0 to 1, leader changed from <none> to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 1 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:43.467394 11606 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 1.
I20240501 14:01:43.470360 11326 catalog_manager.cc:5484] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e reported cstate change: term changed from 0 to 1, leader changed from <none> to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 1 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:43.711851 13187 raft_consensus.cc:482] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:43.712044 13187 raft_consensus.cc:504] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.712589 13187 leader_election.cc:288] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:43.713091 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" is_pre_election: true
I20240501 14:01:43.713145 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e" is_pre_election: true
I20240501 14:01:43.713287 11476 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:43.713389 11867 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 0.
I20240501 14:01:43.713686 11543 leader_election.cc:302] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:43.713953 13187 raft_consensus.cc:2781] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:43.714043 13187 raft_consensus.cc:482] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:43.714136 13187 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.716095 13187 raft_consensus.cc:504] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.716725 13187 leader_election.cc:288] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:43.717078 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:43.717209 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:43.717301 11476 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.717394 11867 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:43.719978 11867 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:43.719978 11476 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 1.
I20240501 14:01:43.720417 11543 leader_election.cc:302] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:43.720650 13187 raft_consensus.cc:2781] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:43.720813 13187 raft_consensus.cc:686] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:43.721108 13187 consensus_queue.cc:237] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:43.723929 11334 catalog_manager.cc:5484] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 1 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:43.854189 12621 consensus_queue.cc:1035] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:43.862390 13232 consensus_queue.cc:1035] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:43.864825 12621 consensus_queue.cc:1035] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:43.892920 13107 consensus_queue.cc:1035] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:44.137671 13133 consensus_queue.cc:1035] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:44.143385 13238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240501 14:01:44.143630 13133 consensus_queue.cc:1035] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:44.143862 13238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:44.155628 13238 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 282.217 ms
time per row: 0.141108 ms
I20240501 14:01:44.600258 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.5:33957
--local_ip_for_outbound_sockets=127.28.17.5
--tserver_master_addrs=127.28.17.62:37567
--webserver_port=35179
--webserver_interface=127.28.17.5
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:01:44.950711 13307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:44.951033 13307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:44.951169 13307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:44.959599 13307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:44.959744 13307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:44.959890 13307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.5
I20240501 14:01:44.969434 13307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.5:33957
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.17.5
--webserver_port=35179
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.5
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:44.970799 13307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:44.972528 13307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:44.985380 13313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:44.985826 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:44.986310 13315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:44.986841 13307 server_base.cc:1034] running on GCE node
I20240501 14:01:44.987476 13307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:44.988480 13307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:44.989703 13307 hybrid_clock.cc:648] HybridClock initialized: now 1714572104989656 us; error 40 us; skew 500 ppm
I20240501 14:01:44.990032 13307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:44.992451 13307 webserver.cc:460] Webserver started at http://127.28.17.5:35179/ using document root <none> and password file <none>
I20240501 14:01:44.993217 13307 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:44.993350 13307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:44.998993 13307 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:01:45.001726 13321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.003283 13307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:01:45.003443 13307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4de87cf6c87c429e8bd46a179d625e8c"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:45.004310 13307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:45.021071 13307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:45.021891 13307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:45.022214 13307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:45.026966 13307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:45.028925 13307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:45.029012 13307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.029136 13307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:45.029187 13307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.065482 13307 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.5:33957
I20240501 14:01:45.065521 13433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.5:33957 every 8 connection(s)
I20240501 14:01:45.067059 13307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:01:45.071439 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 13307
I20240501 14:01:45.072008 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.6:42901
--local_ip_for_outbound_sockets=127.28.17.6
--tserver_master_addrs=127.28.17.62:37567
--webserver_port=34589
--webserver_interface=127.28.17.6
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:45.081055 13434 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:45.081422 13434 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:45.082419 13434 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:45.083994 11333 ts_manager.cc:194] Re-registered known tserver with Master: 4de87cf6c87c429e8bd46a179d625e8c (127.28.17.5:33957)
I20240501 14:01:45.084836 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.5:43347
W20240501 14:01:45.424645 13438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:45.424962 13438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:45.425099 13438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:45.432775 13438 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:45.432946 13438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:45.433130 13438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.6
I20240501 14:01:45.442946 13438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.6:42901
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.17.6
--webserver_port=34589
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.6
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:45.444303 13438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:45.446290 13438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:45.459151 13445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:45.459568 13444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:45.459964 13447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:45.460626 13438 server_base.cc:1034] running on GCE node
I20240501 14:01:45.461237 13438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:45.462216 13438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:45.463419 13438 hybrid_clock.cc:648] HybridClock initialized: now 1714572105463334 us; error 76 us; skew 500 ppm
I20240501 14:01:45.463725 13438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:45.465967 13438 webserver.cc:460] Webserver started at http://127.28.17.6:34589/ using document root <none> and password file <none>
I20240501 14:01:45.466755 13438 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:45.466877 13438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:45.472851 13438 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 14:01:45.475809 13453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.477283 13438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:01:45.477429 13438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "65256cc016354123aab3c93a4f191d0f"
format_stamp: "Formatted at 2024-05-01 14:01:26 on dist-test-slave-dx8c"
I20240501 14:01:45.478228 13438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:45.496589 13438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:45.497454 13438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:45.497716 13438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:45.498726 13438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:45.500367 13438 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:45.500439 13438 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.500551 13438 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:45.500595 13438 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.540282 13438 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.6:42901
I20240501 14:01:45.540316 13565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.6:42901 every 8 connection(s)
I20240501 14:01:45.541651 13438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:01:45.546613 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 13438
I20240501 14:01:45.547282 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.7:46081
--local_ip_for_outbound_sockets=127.28.17.7
--tserver_master_addrs=127.28.17.62:37567
--webserver_port=45671
--webserver_interface=127.28.17.7
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:45.556793 13566 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:45.557179 13566 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:45.558224 13566 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:45.560607 11333 ts_manager.cc:194] Re-registered known tserver with Master: 65256cc016354123aab3c93a4f191d0f (127.28.17.6:42901)
I20240501 14:01:45.561676 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.6:44195
W20240501 14:01:45.889768 13569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:45.890121 13569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:45.890266 13569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:45.898414 13569 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:45.898604 13569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:45.898758 13569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.7
I20240501 14:01:45.908579 13569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.7:46081
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.17.7
--webserver_port=45671
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.7
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:45.909931 13569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:45.911763 13569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:45.925944 13577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:45.926173 13575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:45.926739 13574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:45.927273 13569 server_base.cc:1034] running on GCE node
I20240501 14:01:45.927884 13569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:45.928895 13569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:45.930164 13569 hybrid_clock.cc:648] HybridClock initialized: now 1714572105930120 us; error 41 us; skew 500 ppm
I20240501 14:01:45.930534 13569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:45.933027 13569 webserver.cc:460] Webserver started at http://127.28.17.7:45671/ using document root <none> and password file <none>
I20240501 14:01:45.933920 13569 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:45.934080 13569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:45.940449 13569 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 14:01:45.943554 13583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.945118 13569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:01:45.945294 13569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9e51845ef9224a8bb157cb19425907e1"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:45.946142 13569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:45.969595 13569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:45.970527 13569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:45.970815 13569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:45.971921 13569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:45.973687 13569 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:45.973764 13569 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:45.973860 13569 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:45.973902 13569 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:46.005765 13569 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.7:46081
I20240501 14:01:46.005822 13695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.7:46081 every 8 connection(s)
I20240501 14:01:46.007361 13569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:01:46.013628 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 13569
I20240501 14:01:46.014358 28740 external_mini_cluster.cc:1326] Running /tmp/dist-test-task84tCje/build/asan/bin/kudu
/tmp/dist-test-task84tCje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.17.8:36305
--local_ip_for_outbound_sockets=127.28.17.8
--tserver_master_addrs=127.28.17.62:37567
--webserver_port=44077
--webserver_interface=127.28.17.8
--builtin_ntp_servers=127.28.17.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:01:46.030055 13696 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:46.030537 13696 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:46.031664 13696 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:46.033470 11333 ts_manager.cc:194] Re-registered known tserver with Master: 9e51845ef9224a8bb157cb19425907e1 (127.28.17.7:46081)
I20240501 14:01:46.034584 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.7:58131
I20240501 14:01:46.087796 13434 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
W20240501 14:01:46.371574 13699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:01:46.371891 13699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:01:46.372048 13699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:01:46.380780 13699 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:01:46.380916 13699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:01:46.381045 13699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.17.8
I20240501 14:01:46.390671 13699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.17.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.17.8:36305
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.17.8
--webserver_port=44077
--tserver_master_addrs=127.28.17.62:37567
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.17.8
--log_dir=/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:01:46.392091 13699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:01:46.393898 13699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:46.406937 13705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:46.407255 13707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:46.407802 13704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:46.408303 13699 server_base.cc:1034] running on GCE node
I20240501 14:01:46.409016 13699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:01:46.410091 13699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:01:46.411311 13699 hybrid_clock.cc:648] HybridClock initialized: now 1714572106411249 us; error 56 us; skew 500 ppm
I20240501 14:01:46.411628 13699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:46.413810 13699 webserver.cc:460] Webserver started at http://127.28.17.8:44077/ using document root <none> and password file <none>
I20240501 14:01:46.414574 13699 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:46.414695 13699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:46.420092 13699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 14:01:46.422986 13713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:46.424299 13699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:01:46.424453 13699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6faa12d59f294fc39709e31aa1826aa5"
format_stamp: "Formatted at 2024-05-01 14:01:27 on dist-test-slave-dx8c"
I20240501 14:01:46.425268 13699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:46.481884 13699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:46.482812 13699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:01:46.483081 13699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:46.484038 13699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:46.485553 13699 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:46.485630 13699 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:46.485746 13699 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:46.485797 13699 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:46.516160 13699 rpc_server.cc:307] RPC server started. Bound to: 127.28.17.8:36305
I20240501 14:01:46.516193 13826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.17.8:36305 every 8 connection(s)
I20240501 14:01:46.517665 13699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task84tCje/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714571952612150-28740-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:01:46.523044 28740 external_mini_cluster.cc:1388] Started /tmp/dist-test-task84tCje/build/asan/bin/kudu as pid 13699
I20240501 14:01:46.532999 13827 heartbeater.cc:347] Connected to a master server at 127.28.17.62:37567
I20240501 14:01:46.533368 13827 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:46.534408 13827 heartbeater.cc:511] Master 127.28.17.62:37567 requested a full tablet report, sending...
I20240501 14:01:46.536312 11333 ts_manager.cc:194] Re-registered known tserver with Master: 6faa12d59f294fc39709e31aa1826aa5 (127.28.17.8:36305)
I20240501 14:01:46.537518 11333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.17.8:49199
I20240501 14:01:46.564949 13566 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
I20240501 14:01:46.605388 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.605652 11606 raft_consensus.cc:482] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.605763 11606 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.607720 11606 raft_consensus.cc:504] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.608418 11606 leader_election.cc:288] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.608892 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.609027 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.609135 11866 raft_consensus.cc:3032] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.609210 11735 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.609230 11866 raft_consensus.cc:729] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:46.609529 11866 consensus_queue.cc:260] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.610077 11866 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.611863 11735 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.612418 11540 leader_election.cc:302] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.612717 13290 raft_consensus.cc:2781] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.613062 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.613281 11606 raft_consensus.cc:482] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.613404 11606 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.615038 11866 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.615635 13290 raft_consensus.cc:686] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.615954 13290 consensus_queue.cc:237] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.617889 11606 raft_consensus.cc:504] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.618966 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.619185 11735 raft_consensus.cc:3032] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.619272 11735 raft_consensus.cc:729] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:46.619550 11735 consensus_queue.cc:260] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.619964 11735 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.621698 11735 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.622216 11540 leader_election.cc:302] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.622786 11606 leader_election.cc:288] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.623310 13290 raft_consensus.cc:2781] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.623474 13290 raft_consensus.cc:686] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.623735 13290 consensus_queue.cc:237] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.624069 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b44daf649544443944f1b54859ec381"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.624269 11606 raft_consensus.cc:465] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.624421 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.624624 11866 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.625356 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.625553 11606 raft_consensus.cc:465] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.625425 11333 catalog_manager.cc:5484] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.626575 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.626765 11606 raft_consensus.cc:482] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.626863 11606 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.627328 11866 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.630767 11606 raft_consensus.cc:504] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.631636 11606 leader_election.cc:288] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.632041 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.632277 11735 raft_consensus.cc:3032] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.632360 11735 raft_consensus.cc:729] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:46.632428 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.632616 11866 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.632656 11735 consensus_queue.cc:260] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.632984 11735 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.633087 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.633276 11606 raft_consensus.cc:465] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.634222 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.634390 11607 raft_consensus.cc:482] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.634538 11607 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.635564 11735 raft_consensus.cc:2454] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.635835 11866 raft_consensus.cc:2454] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.636021 11540 leader_election.cc:302] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.636598 13840 raft_consensus.cc:2781] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.636880 13840 raft_consensus.cc:686] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.637292 13840 consensus_queue.cc:237] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.637465 11607 raft_consensus.cc:504] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.638625 11607 leader_election.cc:288] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.639310 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.639569 11735 raft_consensus.cc:3032] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.639662 11735 raft_consensus.cc:729] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:46.639747 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.639974 11607 raft_consensus.cc:482] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.640081 11607 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.639948 11735 consensus_queue.cc:260] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.640323 11735 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.642691 11607 raft_consensus.cc:504] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.643220 11735 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.643661 11540 leader_election.cc:302] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.643829 11607 leader_election.cc:288] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.644013 13840 raft_consensus.cc:2781] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.644059 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.644291 11866 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.644485 13840 raft_consensus.cc:686] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.644758 13840 consensus_queue.cc:237] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.646456 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.646812 11606 raft_consensus.cc:465] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.646840 11866 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.647625 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.647781 11606 raft_consensus.cc:482] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.647809 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:46.647895 11606 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.648023 11476 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.649552 11606 raft_consensus.cc:504] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.650017 11606 leader_election.cc:288] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.650570 11476 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.650775 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.650965 11606 raft_consensus.cc:482] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.651087 11606 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.651440 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.651654 11866 raft_consensus.cc:3032] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.651736 11866 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:46.652091 11866 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.652452 11866 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.653755 11606 raft_consensus.cc:504] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.654759 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.654978 11735 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.656617 11866 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.657263 11543 leader_election.cc:302] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.657331 11735 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.657819 11540 leader_election.cc:302] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.657953 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:46.658097 13290 raft_consensus.cc:2781] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.658157 11476 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.658236 13290 raft_consensus.cc:686] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.658296 13839 raft_consensus.cc:2781] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.658540 13290 consensus_queue.cc:237] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.659955 11333 catalog_manager.cc:5484] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.660543 11333 catalog_manager.cc:5484] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.661003 11333 catalog_manager.cc:5484] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.662420 11476 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.663269 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.663506 11867 raft_consensus.cc:3032] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.663592 11867 raft_consensus.cc:729] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:46.663874 11867 consensus_queue.cc:260] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.664256 11867 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.664247 11606 leader_election.cc:288] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:46.665318 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.666136 11606 raft_consensus.cc:465] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.667145 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.667337 11607 raft_consensus.cc:465] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.667407 11867 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.668215 11543 leader_election.cc:302] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.668889 13840 raft_consensus.cc:2781] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.669027 13840 raft_consensus.cc:686] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.668936 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:46.669173 11474 raft_consensus.cc:3032] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.669265 11474 raft_consensus.cc:729] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.669309 13840 consensus_queue.cc:237] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.669588 11474 consensus_queue.cc:260] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.670152 11474 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.673161 11474 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.674041 13839 raft_consensus.cc:686] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.674388 13839 consensus_queue.cc:237] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.674609 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.674841 11606 raft_consensus.cc:465] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.684412 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.684605 11606 raft_consensus.cc:465] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.685703 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.685902 11606 raft_consensus.cc:465] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.686921 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.687112 11606 raft_consensus.cc:482] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.687240 11606 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.690186 11606 raft_consensus.cc:504] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.692378 11606 leader_election.cc:288] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.692857 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.693070 11735 raft_consensus.cc:3032] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.693178 11735 raft_consensus.cc:729] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:46.693475 11606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b316e036f734208a0cfab91895239b1"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.693449 11735 consensus_queue.cc:260] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.693639 11606 raft_consensus.cc:482] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.693727 11606 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.693822 11735 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.694480 11333 catalog_manager.cc:5484] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.695106 11333 catalog_manager.cc:5484] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.696449 11735 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.696473 11606 raft_consensus.cc:504] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.696828 11540 leader_election.cc:302] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.697271 13290 raft_consensus.cc:2781] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.697417 13290 raft_consensus.cc:686] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.697939 13290 consensus_queue.cc:237] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.699028 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.699246 11866 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.699463 11606 leader_election.cc:288] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.699896 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.700088 11735 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.700668 11605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.700819 11605 raft_consensus.cc:465] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.701692 11605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.701836 11605 raft_consensus.cc:482] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.701941 11605 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.701925 11866 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.703004 11735 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.703040 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:46.703246 11474 raft_consensus.cc:3032] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.703347 11474 raft_consensus.cc:729] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.703652 11474 consensus_queue.cc:260] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.704063 11474 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.704326 11605 raft_consensus.cc:504] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.704842 11605 leader_election.cc:288] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:46.705245 11540 leader_election.cc:302] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.705793 13851 raft_consensus.cc:2781] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.705773 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.705945 11607 raft_consensus.cc:465] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.705942 13851 raft_consensus.cc:686] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.706759 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.706902 11607 raft_consensus.cc:465] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.706976 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.707195 11867 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.707175 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:46.707365 11476 raft_consensus.cc:3032] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.707442 11476 raft_consensus.cc:729] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.707850 11474 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.707922 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:46.708074 11607 raft_consensus.cc:482] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.708186 11607 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.708335 11476 consensus_queue.cc:260] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.709048 11476 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.709878 11867 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.710522 11543 leader_election.cc:302] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.712580 11476 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.710704 11607 raft_consensus.cc:504] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.713162 13839 raft_consensus.cc:2781] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.713327 13839 raft_consensus.cc:686] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.713680 11607 leader_election.cc:288] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.714537 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:46.714740 11474 raft_consensus.cc:3032] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.714814 11474 raft_consensus.cc:729] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.714744 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.714990 11737 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.715075 11474 consensus_queue.cc:260] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.715407 11474 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.717629 11737 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.718020 11474 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 2.
I20240501 14:01:46.713624 13839 consensus_queue.cc:237] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.710667 13851 consensus_queue.cc:237] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.718839 11333 catalog_manager.cc:5484] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.719406 11333 catalog_manager.cc:5484] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.718415 11543 leader_election.cc:302] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.722478 13290 raft_consensus.cc:2781] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.722700 13290 raft_consensus.cc:686] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.723045 13290 consensus_queue.cc:237] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.741528 11333 catalog_manager.cc:5484] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.742146 11333 catalog_manager.cc:5484] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.743094 11333 catalog_manager.cc:5484] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 1 to 2, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 2 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.754583 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b44daf649544443944f1b54859ec381"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.754905 11474 raft_consensus.cc:482] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.755048 11474 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.757964 11474 raft_consensus.cc:504] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.758904 11474 leader_election.cc:288] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.759487 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.759469 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.759688 11866 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.759709 11607 raft_consensus.cc:3032] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.759789 11607 raft_consensus.cc:729] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.760089 11607 consensus_queue.cc:260] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.760541 11607 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.762466 11607 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.762486 11866 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.763069 11411 leader_election.cc:302] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.763439 13439 raft_consensus.cc:2781] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.763584 13439 raft_consensus.cc:686] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.763914 13439 consensus_queue.cc:237] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.764007 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.764417 11474 raft_consensus.cc:465] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.766127 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.766287 11474 raft_consensus.cc:482] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.766371 11474 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.767073 11333 catalog_manager.cc:5484] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.768240 11474 raft_consensus.cc:504] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.768812 11474 leader_election.cc:288] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.769387 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.769611 11607 raft_consensus.cc:3032] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.769701 11607 raft_consensus.cc:729] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.770000 11607 consensus_queue.cc:260] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.770278 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6c91578d215423c88f8759871a591ca"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.770385 11607 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.770466 11474 raft_consensus.cc:465] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.770711 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.770944 11866 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.771307 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.771474 11474 raft_consensus.cc:482] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.771562 11474 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.773115 11607 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.773525 11411 leader_election.cc:302] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.773705 11866 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.773845 13439 raft_consensus.cc:2781] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.774034 13439 raft_consensus.cc:686] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.774204 11474 raft_consensus.cc:504] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.774343 13439 consensus_queue.cc:237] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.775441 11474 leader_election.cc:288] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.776008 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.776293 11735 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.777639 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.777845 11607 raft_consensus.cc:3032] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.777915 11607 raft_consensus.cc:729] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.778174 11607 consensus_queue.cc:260] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.779120 11735 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.779357 11607 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.779678 11409 leader_election.cc:302] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.780084 13855 raft_consensus.cc:2781] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.780828 13855 raft_consensus.cc:686] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.781163 13855 consensus_queue.cc:237] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.781510 11607 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.782541 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.782795 11474 raft_consensus.cc:482] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.783030 11474 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.786026 11474 raft_consensus.cc:504] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.787315 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.787462 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.787595 11607 raft_consensus.cc:3032] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:46.787685 11607 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.787844 11866 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.787995 11607 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.788295 11607 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.788439 11333 catalog_manager.cc:5484] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.791333 11607 raft_consensus.cc:2396] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:46.794171 11866 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 3.
I20240501 14:01:46.794251 11333 catalog_manager.cc:5484] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.794660 11474 leader_election.cc:288] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.794703 11412 leader_election.cc:302] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters: cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:46.795135 13855 raft_consensus.cc:2781] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:46.795284 13855 raft_consensus.cc:686] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.795527 13855 consensus_queue.cc:237] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.795660 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.795866 11476 raft_consensus.cc:482] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.795967 11476 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.798758 11476 raft_consensus.cc:504] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.799551 11476 leader_election.cc:288] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:46.800303 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.800537 11735 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.800899 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.801071 11476 raft_consensus.cc:482] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.801282 11476 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.801692 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.801909 11607 raft_consensus.cc:3032] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.801985 11607 raft_consensus.cc:729] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.802873 11607 consensus_queue.cc:260] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.803252 11607 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.803390 11735 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.804335 11476 raft_consensus.cc:504] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.804628 11409 leader_election.cc:302] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.805076 13439 raft_consensus.cc:2781] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.805212 13439 raft_consensus.cc:686] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.805454 13439 consensus_queue.cc:237] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.805613 11476 leader_election.cc:288] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.806071 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.806155 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.806273 11735 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.806404 11866 raft_consensus.cc:3032] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.806627 11866 raft_consensus.cc:729] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:46.807032 11866 consensus_queue.cc:260] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.807480 11866 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.809568 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.809806 11474 raft_consensus.cc:482] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.809979 11474 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.810129 11735 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.810590 11409 leader_election.cc:302] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.810894 13863 raft_consensus.cc:2781] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.811367 13863 raft_consensus.cc:686] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.811677 13863 consensus_queue.cc:237] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.813244 11474 raft_consensus.cc:504] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.815148 11607 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.816262 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.816473 11867 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.818672 11333 catalog_manager.cc:5484] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 3 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.819118 11867 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.819761 11412 leader_election.cc:302] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:46.819924 11866 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.820148 13855 raft_consensus.cc:2781] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.820318 13855 raft_consensus.cc:686] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.820703 13855 consensus_queue.cc:237] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.820894 11474 leader_election.cc:288] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.822058 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.822280 11735 raft_consensus.cc:3032] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.822381 11735 raft_consensus.cc:729] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:46.822650 11735 consensus_queue.cc:260] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.822944 11735 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.824769 11735 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.825906 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.826121 11474 raft_consensus.cc:482] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.826247 11474 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.828266 11333 catalog_manager.cc:5484] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.828745 11333 catalog_manager.cc:5484] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.828943 11474 raft_consensus.cc:504] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.829115 11333 catalog_manager.cc:5484] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.830233 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.830529 11735 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.830679 11474 leader_election.cc:288] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:46.830888 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.831300 11607 raft_consensus.cc:3032] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.831421 11607 raft_consensus.cc:729] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.831751 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.831760 11607 consensus_queue.cc:260] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.831976 11476 raft_consensus.cc:482] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.832101 11476 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.832340 11607 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.833165 11735 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.833613 11409 leader_election.cc:302] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.833963 13855 raft_consensus.cc:2781] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.834239 13855 raft_consensus.cc:686] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.834672 13855 consensus_queue.cc:237] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.835007 11476 raft_consensus.cc:504] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.836400 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.836584 11606 raft_consensus.cc:3032] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.836529 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.836678 11606 raft_consensus.cc:729] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.836771 11866 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.836943 11606 consensus_queue.cc:260] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.837317 11606 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.839514 11866 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.839736 11606 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.839974 11412 leader_election.cc:302] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:46.840271 11476 leader_election.cc:288] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.840296 13439 raft_consensus.cc:2781] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.840449 13439 raft_consensus.cc:686] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.840680 13439 consensus_queue.cc:237] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.841513 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.841674 11476 raft_consensus.cc:482] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.841797 11476 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.843173 11607 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.844684 11476 raft_consensus.cc:504] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.845700 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.846031 11607 raft_consensus.cc:3032] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.846136 11607 raft_consensus.cc:729] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.846210 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.846422 11866 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.846490 11607 consensus_queue.cc:260] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.846990 11607 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.849176 11866 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.849699 11412 leader_election.cc:302] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:46.850037 13863 raft_consensus.cc:2781] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.850152 11476 leader_election.cc:288] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.850230 13863 raft_consensus.cc:686] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.850677 13863 consensus_queue.cc:237] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.851377 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.851687 11474 raft_consensus.cc:482] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.851814 11474 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.856580 11474 raft_consensus.cc:504] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.858067 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.858371 11866 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.861178 11866 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.862164 11474 leader_election.cc:288] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:46.862845 11412 leader_election.cc:302] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:46.863255 13863 raft_consensus.cc:2781] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.863444 13863 raft_consensus.cc:686] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.862346 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.863669 11606 raft_consensus.cc:3032] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.863801 11606 raft_consensus.cc:729] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.863905 13863 consensus_queue.cc:237] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.864084 11606 consensus_queue.cc:260] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.864509 11606 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.867713 11606 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.868594 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b316e036f734208a0cfab91895239b1"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.868827 11474 raft_consensus.cc:482] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.868970 11474 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.872462 11474 raft_consensus.cc:504] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.873735 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.873998 11735 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.874488 11607 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.876813 11735 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 3.
I20240501 14:01:46.877382 11409 leader_election.cc:302] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.877516 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.877725 11606 raft_consensus.cc:3032] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:46.877843 11606 raft_consensus.cc:729] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.878051 13855 raft_consensus.cc:2781] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:46.878118 11606 consensus_queue.cc:260] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.878232 13855 raft_consensus.cc:686] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.878417 11606 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.878654 13855 consensus_queue.cc:237] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.879014 11333 catalog_manager.cc:5484] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.881130 11606 raft_consensus.cc:2396] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:46.881780 11474 leader_election.cc:288] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:46.883044 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.883244 11476 raft_consensus.cc:482] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.883365 11476 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.885437 11333 catalog_manager.cc:5484] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.886019 11333 catalog_manager.cc:5484] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.886541 11333 catalog_manager.cc:5484] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.887256 11476 raft_consensus.cc:504] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.887676 11475 raft_consensus.cc:1226] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Rejecting Update request from peer 662569f31808430ea81047cac005191e for earlier term 1. Current term is 2. Ops: []
I20240501 14:01:46.888803 13232 consensus_queue.cc:1046] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:46.892978 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.893750 11735 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.894294 11334 catalog_manager.cc:5484] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 3 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.899809 11735 raft_consensus.cc:2454] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.900444 11409 leader_election.cc:302] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.901050 11476 leader_election.cc:288] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.901461 13855 raft_consensus.cc:2781] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.901729 13855 raft_consensus.cc:686] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.902034 13855 consensus_queue.cc:237] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.902305 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.903183 11866 raft_consensus.cc:3032] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.903297 11866 raft_consensus.cc:729] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:46.903546 11866 consensus_queue.cc:260] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.903885 11866 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.905787 11866 raft_consensus.cc:2454] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.906054 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.906304 11474 raft_consensus.cc:482] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.906419 11474 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.908665 11474 raft_consensus.cc:504] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.909838 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.910018 11866 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.912374 11866 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 3.
I20240501 14:01:46.912804 11412 leader_election.cc:302] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:46.912775 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.913005 11607 raft_consensus.cc:3032] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:46.913090 11607 raft_consensus.cc:729] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.913044 11333 catalog_manager.cc:5484] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.913354 11607 consensus_queue.cc:260] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.913646 11607 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.914276 13855 raft_consensus.cc:2781] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:46.914420 13855 raft_consensus.cc:686] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.914688 13855 consensus_queue.cc:237] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
W20240501 14:01:46.915866 13875 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240501 14:01:46.917117 11607 raft_consensus.cc:2396] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:46.917634 11474 leader_election.cc:288] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:46.918725 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.918891 11474 raft_consensus.cc:482] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.919018 11474 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.922022 11474 raft_consensus.cc:504] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.922825 11474 leader_election.cc:288] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:46.923955 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.924175 11866 raft_consensus.cc:3032] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.924252 11866 raft_consensus.cc:729] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:46.924190 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.924403 11735 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.924532 11866 consensus_queue.cc:260] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.924872 11866 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.924738 11333 catalog_manager.cc:5484] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 3 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.927140 11735 raft_consensus.cc:2454] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.927383 11866 raft_consensus.cc:2454] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.928118 11409 leader_election.cc:302] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.928602 13855 raft_consensus.cc:2781] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.928781 13855 raft_consensus.cc:686] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.929023 13855 consensus_queue.cc:237] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:46.930037 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.930948 11474 raft_consensus.cc:482] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.931070 11474 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.932193 11333 catalog_manager.cc:5484] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.934319 11474 raft_consensus.cc:504] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.935453 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.935653 11735 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.937732 11735 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 3.
I20240501 14:01:46.938633 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.938858 11607 raft_consensus.cc:3032] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:46.938963 11607 raft_consensus.cc:729] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.939069 11409 leader_election.cc:302] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.939254 11607 consensus_queue.cc:260] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.939440 13855 raft_consensus.cc:2781] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:46.939548 11607 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.939577 13855 raft_consensus.cc:686] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.940054 13855 consensus_queue.cc:237] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.942234 11607 raft_consensus.cc:2396] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:46.942929 11474 leader_election.cc:288] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:46.944206 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.944409 11476 raft_consensus.cc:465] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 14:01:46.945420 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.945622 11476 raft_consensus.cc:482] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.945742 11476 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.948859 11476 raft_consensus.cc:504] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.950162 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:46.950423 11866 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.952898 11333 catalog_manager.cc:5484] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 3 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.953485 11866 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 3.
I20240501 14:01:46.953697 11476 leader_election.cc:288] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:46.954421 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.954536 11412 leader_election.cc:302] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:46.954766 11607 raft_consensus.cc:3032] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:46.954859 11607 raft_consensus.cc:729] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.955149 11607 consensus_queue.cc:260] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.955430 11607 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.955663 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.955911 11476 raft_consensus.cc:482] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.956125 11476 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.958175 13439 raft_consensus.cc:2781] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:46.958266 11607 raft_consensus.cc:2396] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:46.958343 13439 raft_consensus.cc:686] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.958699 11476 raft_consensus.cc:504] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.958657 13439 consensus_queue.cc:237] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:46.960011 11476 leader_election.cc:288] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.961081 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.961282 11735 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.962421 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.962687 11607 raft_consensus.cc:3032] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:46.962761 11607 raft_consensus.cc:729] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.963021 11607 consensus_queue.cc:260] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.963384 11607 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:46.963039 11333 catalog_manager.cc:5484] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 3 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.963798 11735 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.966142 11607 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 2.
I20240501 14:01:46.966678 11411 leader_election.cc:302] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:46.967039 13439 raft_consensus.cc:2781] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:46.967224 13439 raft_consensus.cc:686] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.967428 11476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:46.967456 13439 consensus_queue.cc:237] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:46.967618 11476 raft_consensus.cc:482] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:46.967732 11476 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.970429 11333 catalog_manager.cc:5484] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 2 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:46.970788 11476 raft_consensus.cc:504] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.971468 11476 leader_election.cc:288] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:46.971908 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:46.972153 11607 raft_consensus.cc:3032] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:46.972229 11607 raft_consensus.cc:729] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:46.972162 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:46.972363 11735 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.972501 11607 consensus_queue.cc:260] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.972776 11607 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:46.975508 11735 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 3.
I20240501 14:01:46.975977 11409 leader_election.cc:302] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:46.976253 13439 raft_consensus.cc:2781] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:46.976416 13439 raft_consensus.cc:686] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:46.976876 13439 consensus_queue.cc:237] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:46.979765 11333 catalog_manager.cc:5484] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 3 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:46.981413 11607 raft_consensus.cc:2396] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:46.999562 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:46.999886 11735 raft_consensus.cc:482] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.000001 11735 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.002074 11735 raft_consensus.cc:504] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.002838 11735 leader_election.cc:288] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.003506 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.003744 11866 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.003968 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.004184 11735 raft_consensus.cc:482] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.004336 11735 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.006662 11735 raft_consensus.cc:504] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.007787 11735 leader_election.cc:288] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.007822 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.008044 11606 raft_consensus.cc:3032] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.008149 11606 raft_consensus.cc:729] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.008421 11606 consensus_queue.cc:260] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.008740 11606 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.010723 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.010944 11867 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.012385 11606 raft_consensus.cc:2396] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.007130 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.013020 11607 raft_consensus.cc:3032] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.013109 11607 raft_consensus.cc:729] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.013674 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.013875 11735 raft_consensus.cc:482] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.013979 11735 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.015988 11867 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.016645 11866 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.017045 11607 consensus_queue.cc:260] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.017360 11607 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.018368 11735 raft_consensus.cc:504] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.020536 11607 raft_consensus.cc:2396] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.022591 11735 leader_election.cc:288] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.023025 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.023123 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.023181 11673 leader_election.cc:302] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters: cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.023244 11476 raft_consensus.cc:3032] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:47.023313 11476 raft_consensus.cc:729] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.023334 11866 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.023623 11476 consensus_queue.cc:260] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.023990 11476 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.026085 11866 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.026986 11476 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.037887 13696 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
I20240501 14:01:47.041476 13887 raft_consensus.cc:2781] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.043464 11673 leader_election.cc:302] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters: cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.048295 13887 raft_consensus.cc:686] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.048723 13887 consensus_queue.cc:237] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.050297 13888 raft_consensus.cc:2781] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.050693 13888 raft_consensus.cc:686] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.051018 13888 consensus_queue.cc:237] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.052125 11673 leader_election.cc:302] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.052999 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6c91578d215423c88f8759871a591ca"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.053179 11735 raft_consensus.cc:482] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.053295 11735 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.056151 11735 raft_consensus.cc:504] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.056864 11735 leader_election.cc:288] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.056928 13888 raft_consensus.cc:2781] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:47.057092 13888 raft_consensus.cc:686] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.057330 13888 consensus_queue.cc:237] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.058274 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.058521 11866 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.058532 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.058771 11476 raft_consensus.cc:3032] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:47.058847 11476 raft_consensus.cc:729] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.059128 11476 consensus_queue.cc:260] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.059476 11476 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.061280 11866 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.062281 11476 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.058537 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.064136 11735 raft_consensus.cc:482] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.064262 11735 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.068382 11333 catalog_manager.cc:5484] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.061929 11673 leader_election.cc:302] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.069356 13889 raft_consensus.cc:2781] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:47.068706 11735 raft_consensus.cc:504] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.071717 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.071942 11607 raft_consensus.cc:3032] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.072017 11607 raft_consensus.cc:729] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.072302 11607 consensus_queue.cc:260] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.072603 11607 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.074728 13889 raft_consensus.cc:686] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.074936 11737 raft_consensus.cc:1261] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Refusing update from remote peer cf637c0b31e34127b7b2599a8c2763a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.075064 13889 consensus_queue.cc:237] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.075862 13290 consensus_queue.cc:1035] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.076131 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.076373 11866 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.076536 11735 leader_election.cc:288] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.078939 11607 raft_consensus.cc:2396] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.079048 11866 raft_consensus.cc:2454] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.079591 11673 leader_election.cc:302] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.079908 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.079980 13888 raft_consensus.cc:2781] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.080121 13888 raft_consensus.cc:686] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.080411 13888 consensus_queue.cc:237] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.081846 11735 raft_consensus.cc:482] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.081992 11735 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.085934 11735 raft_consensus.cc:504] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.086985 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.087213 11607 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.090230 11735 leader_election.cc:288] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.090600 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.090839 11476 raft_consensus.cc:3032] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.090931 11476 raft_consensus.cc:729] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.091226 11476 consensus_queue.cc:260] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.091518 11476 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.092656 11607 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.093570 11672 leader_election.cc:302] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.094420 11476 raft_consensus.cc:2396] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.094702 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.095098 13895 raft_consensus.cc:2781] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.095237 13895 raft_consensus.cc:686] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.095611 13895 consensus_queue.cc:237] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.097959 11735 raft_consensus.cc:482] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.097946 11866 raft_consensus.cc:1261] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Refusing update from remote peer cf637c0b31e34127b7b2599a8c2763a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.098086 11735 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.098735 13839 consensus_queue.cc:1035] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.100139 11334 catalog_manager.cc:5484] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 1 to 2, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 2 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.100710 11334 catalog_manager.cc:5484] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.101217 11334 catalog_manager.cc:5484] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 1 to 2, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 2 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.101363 11735 raft_consensus.cc:504] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.102383 11735 leader_election.cc:288] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.103523 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.103730 11607 raft_consensus.cc:3032] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.103819 11607 raft_consensus.cc:729] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.104079 11607 consensus_queue.cc:260] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.104385 11607 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.104332 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.106791 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.106989 11735 raft_consensus.cc:482] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.107102 11735 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.107581 11607 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.108081 11672 leader_election.cc:302] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters: 662569f31808430ea81047cac005191e
I20240501 14:01:47.108583 13895 raft_consensus.cc:2781] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.108903 13895 raft_consensus.cc:686] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.109309 13895 consensus_queue.cc:237] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.109680 11735 raft_consensus.cc:504] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.110462 11735 leader_election.cc:288] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.110992 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.111217 11476 raft_consensus.cc:3032] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.111317 11476 raft_consensus.cc:729] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.111586 11476 consensus_queue.cc:260] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.111883 11476 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.112716 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.112934 11607 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.114548 11476 raft_consensus.cc:2396] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.115732 11607 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.111889 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.116166 11735 raft_consensus.cc:482] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.116189 11672 leader_election.cc:302] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8
I20240501 14:01:47.116343 11735 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.116734 13895 raft_consensus.cc:2781] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.116894 13895 raft_consensus.cc:686] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.118531 11334 catalog_manager.cc:5484] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.117183 13895 consensus_queue.cc:237] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.119285 11334 catalog_manager.cc:5484] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.119937 11334 catalog_manager.cc:5484] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.120076 11735 raft_consensus.cc:504] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.120702 11735 leader_election.cc:288] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.121793 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.121855 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.122046 11607 raft_consensus.cc:3032] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:47.122121 11607 raft_consensus.cc:729] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.122110 11866 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.122481 11607 consensus_queue.cc:260] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.122879 11607 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.123337 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.123574 11735 raft_consensus.cc:482] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.123732 11735 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.124814 11866 raft_consensus.cc:2454] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.125269 11673 leader_election.cc:302] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.125659 11607 raft_consensus.cc:2454] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.125803 13895 raft_consensus.cc:2781] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:47.126087 13895 raft_consensus.cc:686] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.126420 13895 consensus_queue.cc:237] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.126677 11333 catalog_manager.cc:5484] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.127506 11735 raft_consensus.cc:504] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.128698 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.128968 11476 raft_consensus.cc:3032] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.129057 11476 raft_consensus.cc:729] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.129343 11476 consensus_queue.cc:260] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.129652 11476 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.131484 11735 leader_election.cc:288] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.131979 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.132227 11607 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.132640 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.132830 11735 raft_consensus.cc:482] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.132920 11735 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.133376 11333 catalog_manager.cc:5484] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 2 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.133757 11476 raft_consensus.cc:2396] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.135039 11607 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.135486 11672 leader_election.cc:302] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8
I20240501 14:01:47.135571 11735 raft_consensus.cc:504] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.135969 13895 raft_consensus.cc:2781] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.136207 13895 raft_consensus.cc:686] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.136492 13895 consensus_queue.cc:237] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.136642 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.136853 11476 raft_consensus.cc:3032] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.136925 11476 raft_consensus.cc:729] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.137205 11476 consensus_queue.cc:260] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.136214 11735 leader_election.cc:288] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.137485 11476 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.138568 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.138815 11866 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.138912 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.139078 11735 raft_consensus.cc:482] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.139192 11735 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.140202 11476 raft_consensus.cc:2396] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.141798 11735 raft_consensus.cc:504] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.142792 11867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.143019 11867 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.144645 11866 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.145115 11673 leader_election.cc:302] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8
I20240501 14:01:47.145576 13888 raft_consensus.cc:2781] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.145745 13888 raft_consensus.cc:686] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.145781 11867 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.146054 13888 consensus_queue.cc:237] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.146454 11673 leader_election.cc:302] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.147328 13889 raft_consensus.cc:2781] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.147487 13889 raft_consensus.cc:686] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.147487 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.147696 11476 raft_consensus.cc:3032] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.147764 11476 raft_consensus.cc:729] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.147737 13889 consensus_queue.cc:237] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.148016 11476 consensus_queue.cc:260] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.148281 11476 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.152048 11476 raft_consensus.cc:2396] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.152294 11735 leader_election.cc:288] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.153332 11333 catalog_manager.cc:5484] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.153703 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.153909 11735 raft_consensus.cc:482] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.154021 11735 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.156692 11735 raft_consensus.cc:504] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.157394 11735 leader_election.cc:288] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.157881 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.157892 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.158105 11607 raft_consensus.cc:3032] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:47.158126 11866 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.158210 11607 raft_consensus.cc:729] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.158330 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.158532 11735 raft_consensus.cc:482] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.158454 11607 consensus_queue.cc:260] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.158645 11735 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.158887 11607 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.160822 11866 raft_consensus.cc:2454] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.161281 11673 leader_election.cc:302] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.161433 11607 raft_consensus.cc:2454] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.161612 13888 raft_consensus.cc:2781] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:47.161775 13888 raft_consensus.cc:686] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.161701 11735 raft_consensus.cc:504] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.162081 13888 consensus_queue.cc:237] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.162781 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.163057 11866 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.164453 11333 catalog_manager.cc:5484] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.165403 11333 catalog_manager.cc:5484] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.166596 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.166826 11476 raft_consensus.cc:3032] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.166899 11476 raft_consensus.cc:729] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.167189 11476 consensus_queue.cc:260] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.167476 11476 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.170321 11476 raft_consensus.cc:2396] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.170488 11607 raft_consensus.cc:1261] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.170985 11735 leader_election.cc:288] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.171376 13439 consensus_queue.cc:1035] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.171563 11866 raft_consensus.cc:2454] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.172345 11673 leader_election.cc:302] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8
I20240501 14:01:47.172426 11737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.172648 11737 raft_consensus.cc:482] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.172775 11737 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.175964 11737 raft_consensus.cc:504] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.177302 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.177527 11866 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.178586 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.178835 11605 raft_consensus.cc:3032] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.178910 11605 raft_consensus.cc:729] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.179209 11605 consensus_queue.cc:260] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.179551 11605 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.180385 11866 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.181063 11673 leader_election.cc:302] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.181401 13895 raft_consensus.cc:2781] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.181571 13895 raft_consensus.cc:686] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.181651 13887 raft_consensus.cc:2781] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.181839 13895 consensus_queue.cc:237] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.182441 11605 raft_consensus.cc:2396] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.184474 11737 leader_election.cc:288] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.186064 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b316e036f734208a0cfab91895239b1"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.186362 11735 raft_consensus.cc:482] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.186473 11735 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.189532 11735 raft_consensus.cc:504] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.190907 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.191143 11476 raft_consensus.cc:3032] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.191231 11476 raft_consensus.cc:729] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.191515 11476 consensus_queue.cc:260] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.191820 11476 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.195482 11476 raft_consensus.cc:2396] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.197252 11735 leader_election.cc:288] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.198709 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.199043 11605 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.199293 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.201215 11866 raft_consensus.cc:1261] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.201611 11735 raft_consensus.cc:482] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.199774 11607 raft_consensus.cc:1261] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.201918 13439 consensus_queue.cc:1035] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.202701 11735 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.203703 13439 consensus_queue.cc:1035] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.208986 11605 raft_consensus.cc:2396] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.205462 11735 raft_consensus.cc:504] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.211777 11672 leader_election.cc:302] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.212066 13889 raft_consensus.cc:2726] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.213375 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.213632 11476 raft_consensus.cc:3032] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.213743 11476 raft_consensus.cc:729] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.214128 11476 consensus_queue.cc:260] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.214519 11476 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.217782 11476 raft_consensus.cc:2396] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.218694 13887 raft_consensus.cc:686] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.218685 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.218886 11605 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.219240 11735 leader_election.cc:288] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.219183 13887 consensus_queue.cc:237] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.220249 11737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.220511 11737 raft_consensus.cc:482] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.220685 11737 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.221614 11605 raft_consensus.cc:2396] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.222045 11672 leader_election.cc:302] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.222709 13922 raft_consensus.cc:2726] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.224035 11737 raft_consensus.cc:504] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.225853 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.226128 11866 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.229269 11866 raft_consensus.cc:2454] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.229887 11673 leader_election.cc:302] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.230396 13887 raft_consensus.cc:2781] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.230793 13887 raft_consensus.cc:686] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.231235 11866 raft_consensus.cc:1261] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.231225 13887 consensus_queue.cc:237] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.231916 13439 consensus_queue.cc:1035] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.233832 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.234238 11476 raft_consensus.cc:3032] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.234405 11737 leader_election.cc:288] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.234871 11476 raft_consensus.cc:729] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.238447 11476 consensus_queue.cc:260] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.239312 11476 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.243023 11737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.243364 11737 raft_consensus.cc:482] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.243582 11737 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.246922 11737 raft_consensus.cc:504] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.248436 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.248793 11866 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.254011 11866 raft_consensus.cc:2454] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.254714 11673 leader_election.cc:302] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:47.255219 13887 raft_consensus.cc:2781] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:47.255522 13887 raft_consensus.cc:686] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.255914 13887 consensus_queue.cc:237] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.258121 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.258225 11737 leader_election.cc:288] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.258345 11474 raft_consensus.cc:3032] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:47.258448 11474 raft_consensus.cc:729] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.258832 11474 consensus_queue.cc:260] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.259213 11474 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.259320 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.259519 11735 raft_consensus.cc:482] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.259634 11735 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.260491 11476 raft_consensus.cc:2396] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.262634 11735 raft_consensus.cc:504] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.263623 11474 raft_consensus.cc:2454] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.264017 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.264091 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.264228 11866 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.264379 11735 leader_election.cc:288] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.264360 11606 raft_consensus.cc:3032] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:47.264605 11606 raft_consensus.cc:729] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.265134 11606 consensus_queue.cc:260] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.265697 11606 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:47.266558 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.266758 11735 raft_consensus.cc:482] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.266887 11735 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.268745 11606 raft_consensus.cc:2454] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.269268 11672 leader_election.cc:302] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.269120 11333 catalog_manager.cc:5484] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 2 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.269706 13888 raft_consensus.cc:2781] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:47.269706 11333 catalog_manager.cc:5484] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.269910 13888 raft_consensus.cc:686] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.270144 11866 raft_consensus.cc:2454] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 2.
I20240501 14:01:47.270157 13888 consensus_queue.cc:237] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.270402 11333 catalog_manager.cc:5484] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.272686 11735 raft_consensus.cc:504] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.273347 11735 leader_election.cc:288] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.274019 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.274135 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.274240 11606 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.274338 11474 raft_consensus.cc:3032] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.274433 11474 raft_consensus.cc:729] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.274726 11474 consensus_queue.cc:260] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.275046 11474 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.275424 11866 raft_consensus.cc:1261] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.275430 11735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585"
dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
from {username='slave'} at 127.0.0.1:39908
I20240501 14:01:47.275629 11735 raft_consensus.cc:482] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.275774 11735 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.276613 13855 consensus_queue.cc:1035] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.278165 11474 raft_consensus.cc:2396] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.278096 11606 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd59a6f0c91a434d813bbf69a0b66b6b in term 3.
I20240501 14:01:47.278630 11672 leader_election.cc:302] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8
I20240501 14:01:47.278997 13888 raft_consensus.cc:2781] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.279110 13888 raft_consensus.cc:686] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Leader. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.279359 13888 consensus_queue.cc:237] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.279587 11333 catalog_manager.cc:5484] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.280198 11333 catalog_manager.cc:5484] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 1 to 2, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 2 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.280771 11333 catalog_manager.cc:5484] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 1 to 2, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 2 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.285476 11735 raft_consensus.cc:504] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.286617 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.286849 11606 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.287052 11735 leader_election.cc:288] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 4 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.288108 11333 catalog_manager.cc:5484] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3). New cstate: current_term: 3 leader_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.289649 11606 raft_consensus.cc:2396] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.291357 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.291563 11474 raft_consensus.cc:3032] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.291673 11474 raft_consensus.cc:729] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.291904 11474 consensus_queue.cc:260] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.292245 11474 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.295549 11474 raft_consensus.cc:2396] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cd59a6f0c91a434d813bbf69a0b66b6b for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.296106 11673 leader_election.cc:302] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.297587 13888 raft_consensus.cc:2726] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.297727 11606 raft_consensus.cc:1261] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.297861 11606 pending_rounds.cc:76] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6: Aborting all ops after (but not including) 3
I20240501 14:01:47.297943 11606 pending_rounds.cc:97] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.298077 11606 raft_consensus.cc:2866] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.298781 13855 consensus_queue.cc:1035] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.317452 11866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.317744 11866 raft_consensus.cc:482] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.317876 11866 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.320252 11866 raft_consensus.cc:504] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.321620 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.321815 11606 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.322628 11866 leader_election.cc:288] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.322731 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.322933 11735 raft_consensus.cc:3032] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.323020 11735 raft_consensus.cc:729] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.323279 11735 consensus_queue.cc:260] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.323594 11735 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.323747 11867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.323946 11867 raft_consensus.cc:482] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.324064 11867 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.324479 11606 raft_consensus.cc:2396] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.326325 11735 raft_consensus.cc:2396] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.326939 11867 raft_consensus.cc:504] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.328127 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.328379 11606 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.332176 11800 leader_election.cc:302] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.332652 13875 raft_consensus.cc:2726] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.335500 11606 raft_consensus.cc:2396] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.336036 11867 leader_election.cc:288] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.336694 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.336881 11735 raft_consensus.cc:3032] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.336973 11735 raft_consensus.cc:729] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.337210 11735 consensus_queue.cc:260] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.337460 11735 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.339887 11735 raft_consensus.cc:2396] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.340456 11800 leader_election.cc:302] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.341159 13875 raft_consensus.cc:2726] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.341413 11866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b44daf649544443944f1b54859ec381"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.341625 11866 raft_consensus.cc:482] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.341717 11866 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.344480 11866 raft_consensus.cc:504] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.345795 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.346015 11606 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.348790 11606 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.349282 11802 leader_election.cc:302] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.349632 13875 raft_consensus.cc:2781] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.349817 13875 raft_consensus.cc:686] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.350111 13875 consensus_queue.cc:237] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.351992 11866 leader_election.cc:288] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.352638 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.352854 11474 raft_consensus.cc:3032] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.352981 11474 raft_consensus.cc:729] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.353245 11474 consensus_queue.cc:260] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.353579 11474 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.354368 11866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.354619 11866 raft_consensus.cc:482] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.357384 11474 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.357748 11334 catalog_manager.cc:5484] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.359355 11606 raft_consensus.cc:1261] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.354813 11866 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.360110 13855 consensus_queue.cc:1035] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.362918 11866 raft_consensus.cc:504] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.363708 11866 leader_election.cc:288] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.364101 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.364114 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.364360 11735 raft_consensus.cc:3032] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.364395 11474 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.364468 11735 raft_consensus.cc:729] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.364727 11867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.364749 11735 consensus_queue.cc:260] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.364936 11867 raft_consensus.cc:482] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.365046 11867 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.365065 11735 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.367692 11474 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.367777 11867 raft_consensus.cc:504] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.367916 11735 raft_consensus.cc:2396] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.368518 11867 leader_election.cc:288] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.368990 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.369436 11476 raft_consensus.cc:3032] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.369519 11476 raft_consensus.cc:729] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.369779 11476 consensus_queue.cc:260] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.370127 11476 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.371778 11803 leader_election.cc:302] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.372140 13875 raft_consensus.cc:2781] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.372298 13875 raft_consensus.cc:686] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.372561 13875 consensus_queue.cc:237] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.372828 11476 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.373204 11803 leader_election.cc:302] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:47.373701 13232 raft_consensus.cc:2781] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.373831 13232 raft_consensus.cc:686] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.374080 13232 consensus_queue.cc:237] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.374534 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.374778 11606 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.378031 11606 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.378389 11866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.378729 11866 raft_consensus.cc:482] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.378922 11866 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.387773 11867 raft_consensus.cc:1261] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.388411 13439 consensus_queue.cc:1035] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.389658 11331 catalog_manager.cc:5484] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.392051 11866 raft_consensus.cc:504] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.393182 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.393417 11735 raft_consensus.cc:3032] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.393530 11735 raft_consensus.cc:729] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.393798 11735 consensus_queue.cc:260] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.394096 11735 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.396916 11735 raft_consensus.cc:2396] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.397840 11866 leader_election.cc:288] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.398732 11605 raft_consensus.cc:1261] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.398861 11605 pending_rounds.cc:76] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6: Aborting all ops after (but not including) 3
I20240501 14:01:47.398918 11865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6c91578d215423c88f8759871a591ca"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.399111 11865 raft_consensus.cc:482] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.399210 11865 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.398921 11605 pending_rounds.cc:97] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.399520 11605 raft_consensus.cc:2866] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.400810 13439 consensus_queue.cc:1035] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.401796 11865 raft_consensus.cc:504] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.403034 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.404734 11867 raft_consensus.cc:1261] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.405365 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.405578 11735 raft_consensus.cc:3032] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.405479 13439 consensus_queue.cc:1035] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.405675 11735 raft_consensus.cc:729] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.404438 11865 leader_election.cc:288] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.405982 11332 catalog_manager.cc:5484] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.407879 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.408128 11864 raft_consensus.cc:482] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.408267 11864 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.412777 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.413022 11476 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.413270 11864 raft_consensus.cc:504] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.416760 11476 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.417467 11864 leader_election.cc:288] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.419479 11865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.419478 11735 consensus_queue.cc:260] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.419821 11865 raft_consensus.cc:482] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.419831 11735 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.420017 11865 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.420235 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.420423 11476 raft_consensus.cc:3032] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.420513 11476 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.420858 11476 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.421303 11476 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.405624 11604 raft_consensus.cc:1261] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.421663 11606 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.414345 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.422082 11607 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.422258 13867 consensus_queue.cc:1035] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.424497 11735 raft_consensus.cc:2396] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.424921 11607 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 4.
I20240501 14:01:47.425412 11802 leader_election.cc:302] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.425644 11864 raft_consensus.cc:1261] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.425747 13951 raft_consensus.cc:2781] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.425863 13951 raft_consensus.cc:686] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.426187 11606 raft_consensus.cc:2396] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.426249 13867 consensus_queue.cc:1035] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.426182 13951 consensus_queue.cc:237] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.418138 11605 raft_consensus.cc:1261] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.427927 11802 leader_election.cc:302] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.428378 13875 raft_consensus.cc:2726] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.428812 13867 consensus_queue.cc:1035] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.429229 11865 raft_consensus.cc:504] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.429911 11334 catalog_manager.cc:5484] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 3 to 4, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 4 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.430608 11865 leader_election.cc:288] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.431011 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.431252 11735 raft_consensus.cc:3032] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.431371 11735 raft_consensus.cc:729] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.431645 11735 consensus_queue.cc:260] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.431898 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.431967 11735 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.432129 11864 raft_consensus.cc:482] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.432299 11864 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.433760 11476 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 4.
I20240501 14:01:47.434971 11864 raft_consensus.cc:504] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.435253 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.435505 11606 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.436822 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.437179 11607 raft_consensus.cc:2379] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e in current term 3: Already voted for candidate cd59a6f0c91a434d813bbf69a0b66b6b in this term.
I20240501 14:01:47.438035 11865 raft_consensus.cc:1261] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.439043 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.439327 11737 raft_consensus.cc:2379] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e in current term 3: Already voted for candidate cd59a6f0c91a434d813bbf69a0b66b6b in this term.
I20240501 14:01:47.439832 11800 leader_election.cc:302] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.440171 13875 raft_consensus.cc:2726] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 14:01:47.440840 11606 raft_consensus.cc:2396] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.440863 13867 consensus_queue.cc:1035] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.441419 11803 leader_election.cc:302] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.441737 13875 raft_consensus.cc:2781] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.441886 13875 raft_consensus.cc:686] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.442173 13875 consensus_queue.cc:237] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.442975 11735 raft_consensus.cc:2396] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.448213 11802 leader_election.cc:302] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:47.448561 13875 raft_consensus.cc:2726] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.448249 11864 leader_election.cc:288] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.449731 11606 raft_consensus.cc:1261] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.449867 11606 pending_rounds.cc:76] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6: Aborting all ops after (but not including) 3
I20240501 14:01:47.449921 11606 pending_rounds.cc:97] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.449199 11865 raft_consensus.cc:1261] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Refusing update from remote peer 5cb9d430e4b94d578b436c1f9a9cafb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.450022 11606 raft_consensus.cc:2866] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.450655 13867 consensus_queue.cc:1035] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.450614 11334 catalog_manager.cc:5484] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.452041 13867 consensus_queue.cc:1035] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.460040 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.460408 11864 raft_consensus.cc:482] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.460587 11864 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.489375 11864 raft_consensus.cc:504] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.491575 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.491796 11476 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.494349 13369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.494861 11476 raft_consensus.cc:2396] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.496397 11864 leader_election.cc:288] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.496740 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.497010 11735 raft_consensus.cc:3032] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.497102 11735 raft_consensus.cc:729] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.497370 11735 consensus_queue.cc:260] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.497663 11735 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.497923 11866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.498179 11866 raft_consensus.cc:482] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.498364 11866 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.500591 11735 raft_consensus.cc:2396] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.501044 11800 leader_election.cc:302] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.501490 13875 raft_consensus.cc:2726] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.502535 11866 raft_consensus.cc:504] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.503798 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.504081 11476 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.507057 11476 raft_consensus.cc:2396] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.512234 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.512482 11735 raft_consensus.cc:3032] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.512574 11735 raft_consensus.cc:729] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.512846 11735 consensus_queue.cc:260] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.513147 11735 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.520005 13501 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.523905 11735 raft_consensus.cc:2396] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.524426 11800 leader_election.cc:302] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.524900 13875 raft_consensus.cc:2726] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.531992 11866 leader_election.cc:288] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.533165 11867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.533393 11867 raft_consensus.cc:482] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.533506 11867 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.534853 11864 raft_consensus.cc:1261] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:47.535529 13888 consensus_queue.cc:1035] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:47.536128 11867 raft_consensus.cc:504] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.539188 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.539413 11735 raft_consensus.cc:3032] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.539535 11735 raft_consensus.cc:729] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.543917 11604 raft_consensus.cc:1261] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:47.539790 11735 consensus_queue.cc:260] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.544535 11735 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.545162 13922 consensus_queue.cc:1035] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:47.550968 11867 leader_election.cc:288] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.551661 11604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.551905 11604 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.552170 13827 heartbeater.cc:503] Master 127.28.17.62:37567 was elected leader, sending a full tablet report...
I20240501 14:01:47.552949 11863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.553141 11863 raft_consensus.cc:482] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.553264 11863 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.557502 11863 raft_consensus.cc:504] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.559571 11863 leader_election.cc:288] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.559686 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.559898 11605 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.560647 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.560845 11864 raft_consensus.cc:482] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.560811 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.560940 11864 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.561142 11476 raft_consensus.cc:3032] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.561288 11476 raft_consensus.cc:729] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.561599 11476 consensus_queue.cc:260] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.561996 11476 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.562834 11605 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.562978 11604 raft_consensus.cc:2454] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.563489 11802 leader_election.cc:302] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.563819 11802 leader_election.cc:302] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.564085 13951 raft_consensus.cc:2781] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.564226 13951 raft_consensus.cc:686] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.564146 11864 raft_consensus.cc:504] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.564517 13951 consensus_queue.cc:237] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.565063 11476 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.565429 13951 raft_consensus.cc:2781] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.565363 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.565551 13951 raft_consensus.cc:686] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.565582 11474 raft_consensus.cc:3032] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.565655 11474 raft_consensus.cc:729] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.565804 13951 consensus_queue.cc:237] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.565896 11474 consensus_queue.cc:260] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.566254 11474 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.568697 11474 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.569808 11803 leader_election.cc:302] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:47.569897 11735 raft_consensus.cc:2396] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.570071 13875 raft_consensus.cc:2781] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.570221 13875 raft_consensus.cc:686] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.570463 13875 consensus_queue.cc:237] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.576023 11455 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:01:47.580650 11846 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:01:47.587209 11604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.587478 11604 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.590269 11604 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.591114 11864 leader_election.cc:288] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.593328 11863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.593570 11863 raft_consensus.cc:482] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.593683 11863 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.596938 11863 raft_consensus.cc:504] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.597834 11863 leader_election.cc:288] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.598906 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.599115 11864 raft_consensus.cc:482] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.599241 11864 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.609284 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.609954 11474 raft_consensus.cc:3032] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.610057 11474 raft_consensus.cc:729] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.610402 11474 consensus_queue.cc:260] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.610816 11474 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.611908 11476 raft_consensus.cc:1261] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.612058 11476 pending_rounds.cc:76] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting all ops after (but not including) 3
I20240501 14:01:47.612174 11476 pending_rounds.cc:97] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.612334 11476 raft_consensus.cc:2866] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.613092 13888 consensus_queue.cc:1035] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.613826 11474 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.614334 11803 leader_election.cc:302] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:47.614765 13232 raft_consensus.cc:2781] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.614888 13232 raft_consensus.cc:686] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.615202 13232 consensus_queue.cc:237] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.624334 11715 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:01:47.625805 11864 raft_consensus.cc:504] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.627351 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.627579 11737 raft_consensus.cc:3032] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.627660 11737 raft_consensus.cc:729] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.627812 11474 raft_consensus.cc:1261] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.627920 11474 pending_rounds.cc:76] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting all ops after (but not including) 3
I20240501 14:01:47.628017 11474 pending_rounds.cc:97] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.627915 11737 consensus_queue.cc:260] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.628100 11474 raft_consensus.cc:2866] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.628209 11737 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.628675 13887 consensus_queue.cc:1035] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.640097 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387b254002554b1cb0e41ea5f9092a4d" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.640328 11476 raft_consensus.cc:3037] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.643390 11476 raft_consensus.cc:2396] T 387b254002554b1cb0e41ea5f9092a4d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.644053 11864 leader_election.cc:288] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.644361 11737 raft_consensus.cc:2396] T 387b254002554b1cb0e41ea5f9092a4d P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.645548 11863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.645747 11863 raft_consensus.cc:482] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.645866 11863 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.648555 11863 raft_consensus.cc:504] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.650013 11863 leader_election.cc:288] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.650565 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.650781 11735 raft_consensus.cc:3032] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.650900 11735 raft_consensus.cc:729] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.651155 11735 consensus_queue.cc:260] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.650945 11327 catalog_manager.cc:5484] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.651377 11735 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.651623 11327 catalog_manager.cc:5484] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.657094 11735 raft_consensus.cc:2396] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.657868 11863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.658058 11327 catalog_manager.cc:5484] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.658097 11863 raft_consensus.cc:482] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.658710 11863 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.658664 11327 catalog_manager.cc:5484] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.661942 11800 leader_election.cc:302] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.662302 13951 raft_consensus.cc:2726] T 387b254002554b1cb0e41ea5f9092a4d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.666889 11863 raft_consensus.cc:504] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.667435 11866 raft_consensus.cc:1226] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Rejecting Update request from peer cd59a6f0c91a434d813bbf69a0b66b6b for earlier term 2. Current term is 3. Ops: []
I20240501 14:01:47.668287 13889 consensus_queue.cc:1046] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.668581 13895 raft_consensus.cc:3032] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.668690 13895 raft_consensus.cc:729] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.668999 13895 consensus_queue.cc:260] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.669363 13895 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.669667 11863 leader_election.cc:288] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.670742 11866 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.670943 11866 raft_consensus.cc:482] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.671072 11866 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.672854 11476 raft_consensus.cc:1261] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.672971 11476 pending_rounds.cc:76] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting all ops after (but not including) 3
I20240501 14:01:47.673030 11476 pending_rounds.cc:97] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.673096 11476 raft_consensus.cc:2866] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.673723 13889 consensus_queue.cc:1035] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.673848 11866 raft_consensus.cc:504] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.676272 11866 leader_election.cc:288] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.676522 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.676724 11476 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.677318 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.677515 11864 raft_consensus.cc:482] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.677636 11864 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.677510 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120276baa7874d5aa0351a053598ab0a" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.677845 11735 raft_consensus.cc:3032] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.677978 11735 raft_consensus.cc:729] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.678297 11735 consensus_queue.cc:260] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.678761 11735 raft_consensus.cc:3037] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.679560 11476 raft_consensus.cc:2396] T 120276baa7874d5aa0351a053598ab0a P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.680176 11864 raft_consensus.cc:504] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.675871 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.680544 11737 raft_consensus.cc:2396] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.681139 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.681363 11474 raft_consensus.cc:3032] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.681442 11474 raft_consensus.cc:729] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.681705 11474 consensus_queue.cc:260] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.682089 11474 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.684649 11735 raft_consensus.cc:2396] T 120276baa7874d5aa0351a053598ab0a P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.684950 11474 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 4.
I20240501 14:01:47.685612 11800 leader_election.cc:302] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 662569f31808430ea81047cac005191e; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.686033 11803 leader_election.cc:302] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:47.686239 13951 raft_consensus.cc:2726] T 120276baa7874d5aa0351a053598ab0a P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.686337 13875 raft_consensus.cc:2781] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.686448 13875 raft_consensus.cc:686] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.686719 13875 consensus_queue.cc:237] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.687450 11605 raft_consensus.cc:1261] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.687776 11604 raft_consensus.cc:1261] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.689585 11606 raft_consensus.cc:1261] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.689617 11864 leader_election.cc:288] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.690142 11863 raft_consensus.cc:1261] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:01:47.690748 13895 consensus_queue.cc:1035] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.691761 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.691924 11864 raft_consensus.cc:482] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.692067 11864 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.692211 11327 catalog_manager.cc:5484] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 3 to 4, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 4 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.692740 11606 raft_consensus.cc:1261] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.693212 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.693414 11606 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.694763 11864 raft_consensus.cc:504] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.695984 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.696133 11606 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.696219 11474 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.696614 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.696797 11606 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.697011 11863 raft_consensus.cc:1226] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Rejecting Update request from peer cd59a6f0c91a434d813bbf69a0b66b6b for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20240501 14:01:47.697625 13895 consensus_queue.cc:1046] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.697922 13895 raft_consensus.cc:3032] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:47.697997 13895 raft_consensus.cc:729] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.698266 13895 consensus_queue.cc:260] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.698603 13895 raft_consensus.cc:3037] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.699008 11606 raft_consensus.cc:2454] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.699472 11606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.699539 11474 raft_consensus.cc:2454] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.699524 11604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.699805 11604 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.699832 11606 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:47.700002 11803 leader_election.cc:302] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:47.700304 13875 raft_consensus.cc:2781] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.700455 13875 raft_consensus.cc:686] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.700702 13875 consensus_queue.cc:237] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.702106 11864 leader_election.cc:288] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.702605 11606 raft_consensus.cc:2454] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 3.
I20240501 14:01:47.702836 13895 consensus_queue.cc:1035] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.703446 11604 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 4.
I20240501 14:01:47.704082 11864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41"
dest_uuid: "662569f31808430ea81047cac005191e"
from {username='slave'} at 127.0.0.1:45574
I20240501 14:01:47.704344 11864 raft_consensus.cc:482] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.704521 11864 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.704879 11327 catalog_manager.cc:5484] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.705745 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cecca4823b0a4f1abdfdc3941f5942ee" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.706024 11735 raft_consensus.cc:2396] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 662569f31808430ea81047cac005191e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.707180 13895 consensus_queue.cc:1035] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.707412 11864 raft_consensus.cc:504] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.708051 13895 consensus_queue.cc:1035] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.708312 11864 leader_election.cc:288] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:47.708745 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.708962 11474 raft_consensus.cc:3032] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.709057 11474 raft_consensus.cc:729] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:47.709164 13895 consensus_queue.cc:1035] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.709323 11474 consensus_queue.cc:260] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.709800 11474 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.711319 11802 leader_election.cc:302] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters: cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.711629 13875 raft_consensus.cc:2781] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.711726 11802 leader_election.cc:302] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters: cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.711779 13875 raft_consensus.cc:686] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.712126 13951 raft_consensus.cc:2781] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:47.712039 13875 consensus_queue.cc:237] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.712281 13951 raft_consensus.cc:686] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.712517 13951 consensus_queue.cc:237] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.714766 11474 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 4.
I20240501 14:01:47.715238 11803 leader_election.cc:302] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:47.715497 13232 raft_consensus.cc:2781] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.720098 13762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.720520 13631 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.761230 13232 raft_consensus.cc:686] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Becoming Leader. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.762112 13232 consensus_queue.cc:237] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.763168 11474 raft_consensus.cc:1261] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer cd59a6f0c91a434d813bbf69a0b66b6b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.763274 11474 pending_rounds.cc:76] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting all ops after (but not including) 3
I20240501 14:01:47.763345 11474 pending_rounds.cc:97] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.763414 11474 raft_consensus.cc:2866] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.764154 13888 consensus_queue.cc:1035] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.773736 11327 catalog_manager.cc:5484] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.774413 11327 catalog_manager.cc:5484] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 2 to 3, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 3 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.774636 11604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "662569f31808430ea81047cac005191e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:47.774825 11604 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.776798 11605 raft_consensus.cc:1261] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:47.778466 13875 consensus_queue.cc:1035] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:47.780452 11334 catalog_manager.cc:5484] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e reported cstate change: term changed from 3 to 4, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to 662569f31808430ea81047cac005191e (127.28.17.4). New cstate: current_term: 4 leader_uuid: "662569f31808430ea81047cac005191e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.781456 11604 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 662569f31808430ea81047cac005191e in term 4.
I20240501 14:01:47.792052 11474 raft_consensus.cc:1261] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:47.793872 13232 consensus_queue.cc:1035] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:47.796500 11605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.796758 11605 raft_consensus.cc:482] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.796891 11605 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.799698 11605 raft_consensus.cc:504] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.803318 11605 leader_election.cc:288] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.803417 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.803623 11864 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.803694 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8df7567b084bb691ae69ba715e5d5d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.803892 11735 raft_consensus.cc:3037] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.805191 11605 raft_consensus.cc:1261] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:47.806370 11864 raft_consensus.cc:2454] T 9d8df7567b084bb691ae69ba715e5d5d P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.806870 11543 leader_election.cc:302] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.807184 13290 raft_consensus.cc:2781] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.807338 13290 raft_consensus.cc:686] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.807623 13290 consensus_queue.cc:237] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.807790 11586 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:01:47.808233 13875 consensus_queue.cc:1035] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:47.809716 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.809922 11607 raft_consensus.cc:482] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.810025 11607 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.814270 11607 raft_consensus.cc:504] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.814684 11735 raft_consensus.cc:2396] T 9d8df7567b084bb691ae69ba715e5d5d P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.815501 11607 leader_election.cc:288] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.815964 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.816505 11864 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.816995 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df364f7a0374bdb9933d14a2b72c58e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.818796 11474 raft_consensus.cc:1261] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:01:47.819284 11864 raft_consensus.cc:2454] T 9df364f7a0374bdb9933d14a2b72c58e P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.819525 13951 consensus_queue.cc:1035] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:01:47.819681 11543 leader_election.cc:302] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.820008 13851 raft_consensus.cc:2781] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.828858 11735 raft_consensus.cc:3037] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.832377 11735 raft_consensus.cc:2396] T 9df364f7a0374bdb9933d14a2b72c58e P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.834240 11327 catalog_manager.cc:5484] T 9d8df7567b084bb691ae69ba715e5d5d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 5, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.834743 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b44daf649544443944f1b54859ec381"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.834978 11604 raft_consensus.cc:482] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.835100 11604 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.838234 11604 raft_consensus.cc:504] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.839100 11604 leader_election.cc:288] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.839565 13851 raft_consensus.cc:686] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.840139 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.840157 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.840360 11474 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.840377 11864 raft_consensus.cc:3032] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.840540 11864 raft_consensus.cc:729] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.840529 13851 consensus_queue.cc:237] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.840783 11864 consensus_queue.cc:260] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.841099 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.841164 11864 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.841262 11604 raft_consensus.cc:482] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.841367 11604 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.843159 11474 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.843900 11604 raft_consensus.cc:504] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.844019 11543 leader_election.cc:302] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.844094 11864 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.844561 11604 leader_election.cc:288] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.845336 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.845577 11474 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.845639 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.845871 11604 raft_consensus.cc:482] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.845978 11604 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.846304 13990 raft_consensus.cc:2781] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.846680 13990 raft_consensus.cc:686] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.847223 13990 consensus_queue.cc:237] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.848242 11474 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.848687 11543 leader_election.cc:302] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.848853 11604 raft_consensus.cc:504] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.848984 13990 raft_consensus.cc:2781] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.848999 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.849213 13990 raft_consensus.cc:686] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.849398 11864 raft_consensus.cc:3032] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.849539 11864 raft_consensus.cc:729] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.849666 11604 leader_election.cc:288] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.849900 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.849925 11864 consensus_queue.cc:260] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.850118 11735 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.850457 11864 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.850765 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.850936 11604 raft_consensus.cc:482] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.851063 11604 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.852933 11735 raft_consensus.cc:2396] T a7f0f7056f9449dfb41c885d216102cf P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.853821 11864 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.853824 11604 raft_consensus.cc:504] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.853883 13990 consensus_queue.cc:237] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.854001 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f0f7056f9449dfb41c885d216102cf" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.854185 11863 raft_consensus.cc:3037] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.854909 11604 leader_election.cc:288] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.855166 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.855406 11735 raft_consensus.cc:3032] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.855468 11735 raft_consensus.cc:729] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.855701 11735 consensus_queue.cc:260] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.855736 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.855954 11735 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.856173 11474 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.856753 11863 raft_consensus.cc:2454] T a7f0f7056f9449dfb41c885d216102cf P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.857237 11543 leader_election.cc:302] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters: cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.858024 13851 raft_consensus.cc:2781] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.858099 11735 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.858099 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.858314 13851 raft_consensus.cc:686] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.858310 11604 raft_consensus.cc:482] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.858537 11604 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.858520 11540 leader_election.cc:302] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.858657 13851 consensus_queue.cc:237] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.859133 13290 raft_consensus.cc:2781] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.859292 13290 raft_consensus.cc:686] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.859519 13290 consensus_queue.cc:237] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.859648 11327 catalog_manager.cc:5484] T 9df364f7a0374bdb9933d14a2b72c58e P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 5, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.862279 11476 raft_consensus.cc:1261] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.862273 11604 raft_consensus.cc:504] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.863801 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.864025 11863 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.864027 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69a0e9dd426490393039c70feb27b9d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.864235 11735 raft_consensus.cc:3032] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.864327 11735 raft_consensus.cc:729] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.864578 11735 consensus_queue.cc:260] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.864934 11735 raft_consensus.cc:3037] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.866425 11327 catalog_manager.cc:5484] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.866772 11863 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.866849 13951 consensus_queue.cc:1035] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.867024 11327 catalog_manager.cc:5484] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.867566 11735 raft_consensus.cc:2454] T a69a0e9dd426490393039c70feb27b9d P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.867517 11327 catalog_manager.cc:5484] T a7f0f7056f9449dfb41c885d216102cf P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 5, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.867974 11604 leader_election.cc:288] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.868016 11540 leader_election.cc:302] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.868029 11327 catalog_manager.cc:5484] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.870831 11474 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.875072 13851 raft_consensus.cc:2781] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.875267 13851 raft_consensus.cc:686] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.875622 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.875547 13851 consensus_queue.cc:237] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:47.875787 11604 raft_consensus.cc:482] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.875903 11604 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.879191 11327 catalog_manager.cc:5484] T a69a0e9dd426490393039c70feb27b9d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.880630 11604 raft_consensus.cc:504] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.881402 11604 leader_election.cc:288] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.882093 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.882325 11863 raft_consensus.cc:3032] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:47.882424 11863 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.882529 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.882791 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.883076 11476 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.883754 11474 raft_consensus.cc:1261] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:47.885697 11476 raft_consensus.cc:2454] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.885746 11735 raft_consensus.cc:1261] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.885872 11735 pending_rounds.cc:76] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b: Aborting all ops after (but not including) 3
I20240501 14:01:47.885933 11735 pending_rounds.cc:97] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.886070 11735 raft_consensus.cc:2866] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.886153 11543 leader_election.cc:302] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.886584 13851 raft_consensus.cc:2781] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.886741 13851 raft_consensus.cc:686] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.886974 13982 consensus_queue.cc:1035] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.887043 13851 consensus_queue.cc:237] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.888031 11604 raft_consensus.cc:482] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.888162 11604 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.888985 13982 consensus_queue.cc:1035] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.890283 11863 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.890703 11863 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.891263 11604 raft_consensus.cc:504] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.892419 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.892642 11475 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.892935 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.893133 11735 raft_consensus.cc:3032] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.893200 11735 raft_consensus.cc:729] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.893455 11735 consensus_queue.cc:260] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.893780 11735 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.894304 11863 raft_consensus.cc:2396] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:47.894608 11327 catalog_manager.cc:5484] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 4 to 5, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.895335 11475 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.895792 11543 leader_election.cc:302] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.896054 13851 raft_consensus.cc:2781] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.896198 13851 raft_consensus.cc:686] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.896394 13851 consensus_queue.cc:237] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.898313 11735 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.899030 11604 leader_election.cc:288] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.901746 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.901592 11327 catalog_manager.cc:5484] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.901934 11604 raft_consensus.cc:482] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.902073 11604 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.904881 11604 raft_consensus.cc:504] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.905390 11735 raft_consensus.cc:1261] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:47.905524 11735 pending_rounds.cc:76] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b: Aborting all ops after (but not including) 3
I20240501 14:01:47.905584 11735 pending_rounds.cc:97] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:47.905664 11735 raft_consensus.cc:2866] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:47.905959 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.906040 11604 leader_election.cc:288] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.906180 11474 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.906884 13951 consensus_queue.cc:1035] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:47.906996 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.907188 11604 raft_consensus.cc:482] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.907300 11604 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.908829 11474 raft_consensus.cc:2454] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.909260 11543 leader_election.cc:302] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.909595 13851 raft_consensus.cc:2781] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.909735 13851 raft_consensus.cc:686] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.909991 13851 consensus_queue.cc:237] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.910174 11604 raft_consensus.cc:504] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.910789 11604 leader_election.cc:288] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.911149 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537e656ba83e47d3b990e0b7a7e8e9e2" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.911361 11863 raft_consensus.cc:3032] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:47.911458 11863 raft_consensus.cc:729] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.911701 11863 consensus_queue.cc:260] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.911978 11863 raft_consensus.cc:3037] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.912472 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.912667 11475 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.913537 11327 catalog_manager.cc:5484] T 537e656ba83e47d3b990e0b7a7e8e9e2 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 4 to 5, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.915192 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901d22dfa4fd4cf2b574875ef4f8135b" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.915416 11735 raft_consensus.cc:3037] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.917832 11863 raft_consensus.cc:2396] T 537e656ba83e47d3b990e0b7a7e8e9e2 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:47.918088 11735 raft_consensus.cc:2454] T 901d22dfa4fd4cf2b574875ef4f8135b P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.919675 11475 raft_consensus.cc:2396] T 901d22dfa4fd4cf2b574875ef4f8135b P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.919699 11540 leader_election.cc:302] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.920015 13990 raft_consensus.cc:2781] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.920061 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.920186 13990 raft_consensus.cc:686] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.920248 11604 raft_consensus.cc:482] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.920392 11604 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.920468 13990 consensus_queue.cc:237] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.923174 11604 raft_consensus.cc:504] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.924196 11604 leader_election.cc:288] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.925208 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.925433 11737 raft_consensus.cc:3032] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.925516 11737 raft_consensus.cc:729] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:47.925778 11737 consensus_queue.cc:260] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.926139 11737 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.928231 11737 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.929196 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.929440 11604 raft_consensus.cc:482] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.929556 11604 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.929196 11540 leader_election.cc:302] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.930198 13851 raft_consensus.cc:2781] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.930366 13851 raft_consensus.cc:686] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.930828 13851 consensus_queue.cc:237] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.932356 11604 raft_consensus.cc:504] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.933058 11604 leader_election.cc:288] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.935395 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.935477 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.935639 11474 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.935712 11863 raft_consensus.cc:3032] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.935819 11863 raft_consensus.cc:729] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.935997 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e510660118b4d71a1835c6ffa5a02ed" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.936098 11863 consensus_queue.cc:260] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.936319 11737 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.936403 11863 raft_consensus.cc:3037] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.938323 11474 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.939924 11863 raft_consensus.cc:2396] T 4e510660118b4d71a1835c6ffa5a02ed P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.940099 11737 raft_consensus.cc:2396] T 4e510660118b4d71a1835c6ffa5a02ed P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:47.940667 11540 leader_election.cc:302] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf637c0b31e34127b7b2599a8c2763a6; no voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.941226 13851 raft_consensus.cc:2726] T 4e510660118b4d71a1835c6ffa5a02ed P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:47.942724 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.942934 11604 raft_consensus.cc:482] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.943054 11604 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.942939 11334 catalog_manager.cc:5484] T 901d22dfa4fd4cf2b574875ef4f8135b P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 5, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:47.946456 11604 raft_consensus.cc:504] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.947407 11604 leader_election.cc:288] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.948280 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.948488 11474 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.948475 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.948700 11863 raft_consensus.cc:3032] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.948784 11863 raft_consensus.cc:729] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.948730 11334 catalog_manager.cc:5484] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.949040 11863 consensus_queue.cc:260] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.949343 11863 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.951246 11474 raft_consensus.cc:2454] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.951915 11543 leader_election.cc:302] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.952231 13990 raft_consensus.cc:2781] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.952731 13990 raft_consensus.cc:686] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.953122 11863 raft_consensus.cc:2396] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.953078 13990 consensus_queue.cc:237] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.954663 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.954897 11604 raft_consensus.cc:482] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.955004 11604 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.957828 11334 catalog_manager.cc:5484] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.964290 11604 raft_consensus.cc:504] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.965095 11604 leader_election.cc:288] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.965689 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.965587 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.965883 11863 raft_consensus.cc:3032] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.965925 11474 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.965988 11863 raft_consensus.cc:729] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.966244 11863 consensus_queue.cc:260] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.966550 11863 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.968930 11474 raft_consensus.cc:2454] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.969377 11863 raft_consensus.cc:2396] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.969504 11543 leader_election.cc:302] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.970193 13990 raft_consensus.cc:2781] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.970352 13990 raft_consensus.cc:686] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.970677 13990 consensus_queue.cc:237] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:47.971485 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.971701 11604 raft_consensus.cc:482] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.971853 11604 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.974066 11334 catalog_manager.cc:5484] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.974740 11604 raft_consensus.cc:504] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.975909 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.976102 11863 raft_consensus.cc:3032] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:47.976187 11863 raft_consensus.cc:729] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:47.976438 11863 consensus_queue.cc:260] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.976732 11863 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.978868 11604 leader_election.cc:288] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:47.979413 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.979616 11474 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:47.980186 11863 raft_consensus.cc:2396] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.982149 11474 raft_consensus.cc:2454] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:47.982800 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.983223 11604 raft_consensus.cc:482] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.983345 11604 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.983481 11543 leader_election.cc:302] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters: 662569f31808430ea81047cac005191e
I20240501 14:01:47.983790 13990 raft_consensus.cc:2781] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:47.983947 13990 raft_consensus.cc:686] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.984222 13990 consensus_queue.cc:237] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:47.986405 11604 raft_consensus.cc:504] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.987116 11604 leader_election.cc:288] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:47.987521 11327 catalog_manager.cc:5484] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:47.988941 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.989177 11737 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.988870 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5062be5a545c430b8f9277e77c51e11e" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:47.989322 11863 raft_consensus.cc:3037] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.990375 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b316e036f734208a0cfab91895239b1"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:47.990569 11604 raft_consensus.cc:482] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:47.990691 11604 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.991958 11737 raft_consensus.cc:2396] T 5062be5a545c430b8f9277e77c51e11e P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:47.992971 11863 raft_consensus.cc:2454] T 5062be5a545c430b8f9277e77c51e11e P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.993157 11604 raft_consensus.cc:504] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.993460 11543 leader_election.cc:302] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6; no voters: cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:47.993979 13851 raft_consensus.cc:2781] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.994123 13851 raft_consensus.cc:686] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.994308 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:47.994534 11737 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.994400 13851 consensus_queue.cc:237] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:47.995980 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b316e036f734208a0cfab91895239b1" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:47.996209 11474 raft_consensus.cc:3037] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:47.997059 11737 raft_consensus.cc:2454] T 5b316e036f734208a0cfab91895239b1 P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:47.997480 11540 leader_election.cc:302] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:47.997817 11604 leader_election.cc:288] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:47.998787 13990 raft_consensus.cc:2781] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:47.998934 13990 raft_consensus.cc:686] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:47.999220 13990 consensus_queue.cc:237] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:48.001035 11604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:48.001236 11604 raft_consensus.cc:482] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.001344 11604 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.001489 11474 raft_consensus.cc:2396] T 5b316e036f734208a0cfab91895239b1 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.004350 11604 raft_consensus.cc:504] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.010762 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.011001 11737 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.013736 11737 raft_consensus.cc:2396] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:48.013583 11327 catalog_manager.cc:5484] T 5062be5a545c430b8f9277e77c51e11e P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 5, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:48.015362 11604 leader_election.cc:288] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:48.017274 11605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:48.017496 11605 raft_consensus.cc:482] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.017591 11605 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.019100 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825b8c2dd4504ce7acf0b8f8218ac9ca" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.019326 11863 raft_consensus.cc:3032] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:48.019407 11863 raft_consensus.cc:729] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:48.019671 11863 consensus_queue.cc:260] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.019953 11863 raft_consensus.cc:3037] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.020038 11327 catalog_manager.cc:5484] T 5b316e036f734208a0cfab91895239b1 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 5, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:48.020550 11605 raft_consensus.cc:504] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:48.021168 11605 leader_election.cc:288] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:48.022594 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:48.022768 11607 raft_consensus.cc:482] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.022890 11607 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.025576 11607 raft_consensus.cc:504] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.026186 11607 leader_election.cc:288] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:48.027130 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.027316 11735 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.027604 11863 raft_consensus.cc:2396] T 825b8c2dd4504ce7acf0b8f8218ac9ca P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:48.029922 11735 raft_consensus.cc:2396] T a08f72693051415bb8956a9dd7f43476 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:48.031574 11543 leader_election.cc:302] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf637c0b31e34127b7b2599a8c2763a6; no voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:48.031877 13990 raft_consensus.cc:2726] T 825b8c2dd4504ce7acf0b8f8218ac9ca P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:48.032756 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08f72693051415bb8956a9dd7f43476" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.032756 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:48.032948 11864 raft_consensus.cc:3032] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:48.032948 11474 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.033038 11864 raft_consensus.cc:729] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:48.033309 11864 consensus_queue.cc:260] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.033594 11864 raft_consensus.cc:3037] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.034665 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ce26abfa4142d4a83dbe8705f93b41" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.034878 11863 raft_consensus.cc:3032] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.034948 11863 raft_consensus.cc:729] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:48.035218 11863 consensus_queue.cc:260] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:48.035508 11863 raft_consensus.cc:3037] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.035594 11474 raft_consensus.cc:2454] T 47ce26abfa4142d4a83dbe8705f93b41 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:48.036289 11864 raft_consensus.cc:2396] T a08f72693051415bb8956a9dd7f43476 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 14:01:48.036268 11543 leader_election.cc:302] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:48.036554 13990 raft_consensus.cc:2781] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.036695 13990 raft_consensus.cc:686] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.036981 11607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:48.037271 11607 raft_consensus.cc:482] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.037506 11607 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.037358 11543 leader_election.cc:302] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf637c0b31e34127b7b2599a8c2763a6; no voters: 662569f31808430ea81047cac005191e, cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:48.037951 13290 raft_consensus.cc:2726] T a08f72693051415bb8956a9dd7f43476 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 14:01:48.036948 13990 consensus_queue.cc:237] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:48.040400 11607 raft_consensus.cc:504] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.041275 11863 raft_consensus.cc:2396] T 47ce26abfa4142d4a83dbe8705f93b41 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:48.041416 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.041631 11735 raft_consensus.cc:3032] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:48.041702 11735 raft_consensus.cc:729] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cd59a6f0c91a434d813bbf69a0b66b6b, State: Running, Role: LEADER
I20240501 14:01:48.041473 11334 catalog_manager.cc:5484] T 47ce26abfa4142d4a83dbe8705f93b41 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 4 to 5, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:48.041981 11735 consensus_queue.cc:260] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.042364 11735 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.045116 11735 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:48.045517 11540 leader_election.cc:302] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters:
I20240501 14:01:48.046335 11607 leader_election.cc:288] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 4 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355)
I20240501 14:01:48.046437 13851 raft_consensus.cc:2781] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:48.046614 13851 raft_consensus.cc:686] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.046891 13851 consensus_queue.cc:237] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.047535 11605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585"
dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
from {username='slave'} at 127.0.0.1:39352
I20240501 14:01:48.048048 11605 raft_consensus.cc:482] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.048180 11605 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.049759 11334 catalog_manager.cc:5484] T 2a35808076b34ec49f8897a477fc0b3d P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 4, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 4 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.051064 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a35808076b34ec49f8897a477fc0b3d" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:48.051295 11474 raft_consensus.cc:3037] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.054018 11474 raft_consensus.cc:2454] T 2a35808076b34ec49f8897a477fc0b3d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 4.
I20240501 14:01:48.050720 11605 raft_consensus.cc:504] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.055581 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
I20240501 14:01:48.055801 11475 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.058527 11475 raft_consensus.cc:2396] T 08f0000ac0254bf3be818261a9a1a585 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf637c0b31e34127b7b2599a8c2763a6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.059738 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f0000ac0254bf3be818261a9a1a585" candidate_uuid: "cf637c0b31e34127b7b2599a8c2763a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.059973 11735 raft_consensus.cc:3037] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.062609 11605 leader_election.cc:288] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Requested vote from peers 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1:44355), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:48.063457 11735 raft_consensus.cc:2454] T 08f0000ac0254bf3be818261a9a1a585 P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf637c0b31e34127b7b2599a8c2763a6 in term 5.
I20240501 14:01:48.064406 11540 leader_election.cc:302] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd59a6f0c91a434d813bbf69a0b66b6b, cf637c0b31e34127b7b2599a8c2763a6; no voters: 5cb9d430e4b94d578b436c1f9a9cafb8
I20240501 14:01:48.064792 13851 raft_consensus.cc:2781] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.065008 13851 raft_consensus.cc:686] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Leader. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.065315 13851 consensus_queue.cc:237] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.074390 11475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b44daf649544443944f1b54859ec381"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.074658 11475 raft_consensus.cc:482] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.074790 11475 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.077512 11475 raft_consensus.cc:504] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.078626 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.078850 11605 raft_consensus.cc:3032] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.078943 11605 raft_consensus.cc:729] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.079221 11605 consensus_queue.cc:260] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.079488 11605 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.082707 11475 leader_election.cc:288] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:48.083303 11605 raft_consensus.cc:2396] T 7b44daf649544443944f1b54859ec381 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 14:01:48.083251 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b44daf649544443944f1b54859ec381" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.083474 11863 raft_consensus.cc:3037] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.086066 11863 raft_consensus.cc:2454] T 7b44daf649544443944f1b54859ec381 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 5.
I20240501 14:01:48.086548 11412 leader_election.cc:302] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters: cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:48.086917 13439 raft_consensus.cc:2781] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.087060 13439 raft_consensus.cc:686] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.087363 13439 consensus_queue.cc:237] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.087715 11475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.087903 11475 raft_consensus.cc:482] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.088016 11475 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.091058 11475 raft_consensus.cc:504] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.090883 11334 catalog_manager.cc:5484] T 7b44daf649544443944f1b54859ec381 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 4 to 5, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 5 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.092211 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.092408 11735 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.094949 11735 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 4.
I20240501 14:01:48.095427 11409 leader_election.cc:302] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:48.095600 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b596b92c95d24211973cd278eb57c5ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.095739 13855 raft_consensus.cc:2781] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:48.095870 13855 raft_consensus.cc:686] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.095882 11863 raft_consensus.cc:3032] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:48.096076 11863 raft_consensus.cc:729] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:48.096159 13855 consensus_queue.cc:237] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.096496 11863 consensus_queue.cc:260] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.097121 11475 leader_election.cc:288] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 4 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:48.098379 11475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.098706 11475 raft_consensus.cc:482] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.098877 11475 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.099257 11334 catalog_manager.cc:5484] T b596b92c95d24211973cd278eb57c5ca P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 3 to 4, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 4 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.101416 11475 raft_consensus.cc:504] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.096977 11863 raft_consensus.cc:3037] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.102669 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.102952 11605 raft_consensus.cc:3032] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.103041 11605 raft_consensus.cc:729] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.103307 11605 consensus_queue.cc:260] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.103580 11605 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.105479 11863 raft_consensus.cc:2454] T b596b92c95d24211973cd278eb57c5ca P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 4.
I20240501 14:01:48.106320 11605 raft_consensus.cc:2396] T 658947a6b41844c593994a56bf20d0b0 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 14:01:48.110862 11475 leader_election.cc:288] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:48.112612 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "658947a6b41844c593994a56bf20d0b0" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.112834 11863 raft_consensus.cc:3037] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.113672 11334 catalog_manager.cc:5484] T 08f0000ac0254bf3be818261a9a1a585 P cf637c0b31e34127b7b2599a8c2763a6 reported cstate change: term changed from 3 to 5, leader changed from 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1) to cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2). New cstate: current_term: 5 leader_uuid: "cf637c0b31e34127b7b2599a8c2763a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.115763 11863 raft_consensus.cc:2454] T 658947a6b41844c593994a56bf20d0b0 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 5.
I20240501 14:01:48.116333 11412 leader_election.cc:302] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters: cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:48.116464 11475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6c91578d215423c88f8759871a591ca"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.116678 11475 raft_consensus.cc:482] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.116775 11475 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.116910 13439 raft_consensus.cc:2781] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.117062 13439 raft_consensus.cc:686] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.117336 13439 consensus_queue.cc:237] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.120501 11334 catalog_manager.cc:5484] T 658947a6b41844c593994a56bf20d0b0 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 4 to 5, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 5 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.121342 11475 raft_consensus.cc:504] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.122453 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.122575 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6c91578d215423c88f8759871a591ca" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.122704 11863 raft_consensus.cc:3032] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:48.122776 11863 raft_consensus.cc:729] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 662569f31808430ea81047cac005191e, State: Running, Role: LEADER
I20240501 14:01:48.122797 11735 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.123080 11863 consensus_queue.cc:260] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.123427 11863 raft_consensus.cc:3037] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:48.125947 11735 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 4.
I20240501 14:01:48.126080 11863 raft_consensus.cc:2454] T c6c91578d215423c88f8759871a591ca P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 4.
I20240501 14:01:48.126133 11475 leader_election.cc:288] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 4 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:48.126528 11412 leader_election.cc:302] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:48.126837 13439 raft_consensus.cc:2781] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:48.126981 13439 raft_consensus.cc:686] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.127290 11475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.128129 11475 raft_consensus.cc:482] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.128353 11475 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.127472 13439 consensus_queue.cc:237] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.131094 11475 raft_consensus.cc:504] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.132166 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.132443 11605 raft_consensus.cc:3032] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.132591 11605 raft_consensus.cc:729] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.132933 11605 consensus_queue.cc:260] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.133280 11605 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.134337 11334 catalog_manager.cc:5484] T c6c91578d215423c88f8759871a591ca P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 3 to 4, leader changed from 662569f31808430ea81047cac005191e (127.28.17.4) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 4 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.136250 11605 raft_consensus.cc:2396] T 91002b1d230d442e91be4067b39c8acb P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:48.137372 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91002b1d230d442e91be4067b39c8acb" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.137583 11735 raft_consensus.cc:3037] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.140395 11735 raft_consensus.cc:2454] T 91002b1d230d442e91be4067b39c8acb P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 5.
I20240501 14:01:48.140830 11409 leader_election.cc:302] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters: cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:48.141131 13439 raft_consensus.cc:2781] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.141193 11475 leader_election.cc:288] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:48.141634 13439 raft_consensus.cc:686] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.141914 13439 consensus_queue.cc:237] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.142385 11475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.142647 11475 raft_consensus.cc:482] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.142793 11475 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:48.145670 11475 raft_consensus.cc:504] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.145527 11334 catalog_manager.cc:5484] T 91002b1d230d442e91be4067b39c8acb P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 4 to 5, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 5 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:48.146790 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.147022 11605 raft_consensus.cc:3032] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:01:48.147112 11605 raft_consensus.cc:729] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.147385 11605 consensus_queue.cc:260] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.147667 11605 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:48.150558 11605 raft_consensus.cc:2396] T 7cc63424584f4016967b1e2cf0352ee4 P cf637c0b31e34127b7b2599a8c2763a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:48.150854 11475 leader_election.cc:288] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 6 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:48.151458 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc63424584f4016967b1e2cf0352ee4" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.151710 11863 raft_consensus.cc:3037] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:48.152033 11475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.152330 11475 raft_consensus.cc:482] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.152453 11475 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.153915 11863 raft_consensus.cc:2396] T 7cc63424584f4016967b1e2cf0352ee4 P 662569f31808430ea81047cac005191e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:48.154361 11412 leader_election.cc:302] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8; no voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:48.154644 13439 raft_consensus.cc:2726] T 7cc63424584f4016967b1e2cf0352ee4 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 14:01:48.155154 11475 raft_consensus.cc:504] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:48.156289 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.156421 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "571d6b5b23424ea29dbe787584525efc" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.156514 11735 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.156627 11605 raft_consensus.cc:3032] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.156715 11605 raft_consensus.cc:729] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.157054 11605 consensus_queue.cc:260] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:48.157321 11605 raft_consensus.cc:3037] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.159236 11735 raft_consensus.cc:2454] T 571d6b5b23424ea29dbe787584525efc P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 5.
I20240501 14:01:48.159754 11409 leader_election.cc:302] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:48.160069 13439 raft_consensus.cc:2781] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.160230 13439 raft_consensus.cc:686] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.160542 11735 raft_consensus.cc:1261] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 14:01:48.160497 13439 consensus_queue.cc:237] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } }
I20240501 14:01:48.160634 11735 pending_rounds.cc:76] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b: Aborting all ops after (but not including) 3
I20240501 14:01:48.160698 11735 pending_rounds.cc:97] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 14:01:48.160765 11735 raft_consensus.cc:2866] T cecca4823b0a4f1abdfdc3941f5942ee P cd59a6f0c91a434d813bbf69a0b66b6b [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 14:01:48.161314 13951 consensus_queue.cc:1035] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:48.164016 11605 raft_consensus.cc:2396] T 571d6b5b23424ea29dbe787584525efc P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:48.164027 11334 catalog_manager.cc:5484] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 4 to 5, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 5 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.164530 11475 leader_election.cc:288] T 571d6b5b23424ea29dbe787584525efc P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:48.165675 11474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.165858 11474 raft_consensus.cc:482] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.165967 11474 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.168041 11476 raft_consensus.cc:1261] T cecca4823b0a4f1abdfdc3941f5942ee P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 3 FOLLOWER]: Refusing update from remote peer 662569f31808430ea81047cac005191e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 14:01:48.168560 11474 raft_consensus.cc:504] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.169152 13875 consensus_queue.cc:1035] T cecca4823b0a4f1abdfdc3941f5942ee P 662569f31808430ea81047cac005191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:01:48.169996 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.170182 11735 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.170333 11474 leader_election.cc:288] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:48.171160 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d00ce95a04f49dfa3df17ee8998955c" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.171377 11863 raft_consensus.cc:3037] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.171340 11473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.171504 11473 raft_consensus.cc:482] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.171607 11473 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.172888 11735 raft_consensus.cc:2396] T 8d00ce95a04f49dfa3df17ee8998955c P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.174140 11473 raft_consensus.cc:504] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.174309 11863 raft_consensus.cc:2454] T 8d00ce95a04f49dfa3df17ee8998955c P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 5.
I20240501 14:01:48.175438 11473 leader_election.cc:288] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807)
I20240501 14:01:48.175849 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.176038 11864 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.176043 11412 leader_election.cc:302] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters: cd59a6f0c91a434d813bbf69a0b66b6b
I20240501 14:01:48.176574 13855 raft_consensus.cc:2781] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.176736 13855 raft_consensus.cc:686] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.176708 11472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.176934 11472 raft_consensus.cc:482] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.177057 11472 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.177016 13855 consensus_queue.cc:237] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.178637 11864 raft_consensus.cc:2454] T 35930481673f43e2bd8f29ed2ca5b317 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 5.
I20240501 14:01:48.179523 11412 leader_election.cc:302] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, 662569f31808430ea81047cac005191e; no voters:
I20240501 14:01:48.179823 13855 raft_consensus.cc:2781] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.179770 11472 raft_consensus.cc:504] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.179925 13855 raft_consensus.cc:686] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.180212 13855 consensus_queue.cc:237] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } }
I20240501 14:01:48.180805 11737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.180871 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a9c838c4304ff0823e9cffff09bc58" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.181027 11737 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.181075 11607 raft_consensus.cc:3032] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.181169 11607 raft_consensus.cc:729] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.181416 11607 consensus_queue.cc:260] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.181717 11607 raft_consensus.cc:3037] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.182808 11472 leader_election.cc:288] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3:37807), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:48.184111 11737 raft_consensus.cc:2454] T 63a9c838c4304ff0823e9cffff09bc58 P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb9d430e4b94d578b436c1f9a9cafb8 in term 5.
I20240501 14:01:48.184693 11409 leader_election.cc:302] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8, cd59a6f0c91a434d813bbf69a0b66b6b; no voters:
I20240501 14:01:48.185724 11607 raft_consensus.cc:2396] T 63a9c838c4304ff0823e9cffff09bc58 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 14:01:48.186018 11472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.186298 11472 raft_consensus.cc:482] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.186465 11472 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.186718 11735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35930481673f43e2bd8f29ed2ca5b317" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b"
I20240501 14:01:48.186925 11735 raft_consensus.cc:3037] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.186512 13439 raft_consensus.cc:2781] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:48.187430 13439 raft_consensus.cc:686] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 LEADER]: Becoming Leader. State: Replica: 5cb9d430e4b94d578b436c1f9a9cafb8, State: Running, Role: LEADER
I20240501 14:01:48.187709 13439 consensus_queue.cc:237] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.191690 11735 raft_consensus.cc:2396] T 35930481673f43e2bd8f29ed2ca5b317 P cd59a6f0c91a434d813bbf69a0b66b6b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.190380 11472 raft_consensus.cc:504] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.194198 11472 leader_election.cc:288] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:48.194960 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.195124 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf124b7d872c4c2aacddf2a2ea08374d" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.195210 11605 raft_consensus.cc:3032] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.195284 11605 raft_consensus.cc:729] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.195340 11863 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.195561 11605 consensus_queue.cc:260] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.195878 11605 raft_consensus.cc:3037] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.198102 11863 raft_consensus.cc:2396] T cf124b7d872c4c2aacddf2a2ea08374d P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.197901 11327 catalog_manager.cc:5484] T 8d00ce95a04f49dfa3df17ee8998955c P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 3 to 5, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 5 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.198652 11605 raft_consensus.cc:2396] T cf124b7d872c4c2aacddf2a2ea08374d P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.199086 11411 leader_election.cc:302] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8; no voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:48.199585 14021 raft_consensus.cc:2726] T cf124b7d872c4c2aacddf2a2ea08374d P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 14:01:48.200738 11472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.200934 11472 raft_consensus.cc:482] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.201064 11472 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.204320 11472 raft_consensus.cc:504] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.204470 11327 catalog_manager.cc:5484] T 63a9c838c4304ff0823e9cffff09bc58 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 4 to 5, leader changed from cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 5 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:48.205066 11472 leader_election.cc:288] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357), 662569f31808430ea81047cac005191e (127.28.17.4:33167)
I20240501 14:01:48.205060 11327 catalog_manager.cc:5484] T 35930481673f43e2bd8f29ed2ca5b317 P 5cb9d430e4b94d578b436c1f9a9cafb8 reported cstate change: term changed from 3 to 5, leader changed from cd59a6f0c91a434d813bbf69a0b66b6b (127.28.17.3) to 5cb9d430e4b94d578b436c1f9a9cafb8 (127.28.17.1). New cstate: current_term: 5 leader_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd59a6f0c91a434d813bbf69a0b66b6b" member_type: VOTER last_known_addr { host: "127.28.17.3" port: 37807 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:48.205513 11863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.205710 11863 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.205464 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2563e19394b1476d89a4bafa24aa90c6" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.205899 11605 raft_consensus.cc:3032] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.206001 11605 raft_consensus.cc:729] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.206297 11605 consensus_queue.cc:260] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } }
I20240501 14:01:48.206401 11472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'} at 127.0.0.1:46310
I20240501 14:01:48.206604 11605 raft_consensus.cc:3037] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.206656 11472 raft_consensus.cc:482] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:48.206755 11472 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.208535 11863 raft_consensus.cc:2396] T 2563e19394b1476d89a4bafa24aa90c6 P 662569f31808430ea81047cac005191e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.209493 11605 raft_consensus.cc:2396] T 2563e19394b1476d89a4bafa24aa90c6 P cf637c0b31e34127b7b2599a8c2763a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5cb9d430e4b94d578b436c1f9a9cafb8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 14:01:48.209556 11472 raft_consensus.cc:504] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.210436 11412 leader_election.cc:302] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5cb9d430e4b94d578b436c1f9a9cafb8; no voters: 662569f31808430ea81047cac005191e, cf637c0b31e34127b7b2599a8c2763a6
I20240501 14:01:48.210741 14021 raft_consensus.cc:2726] T 2563e19394b1476d89a4bafa24aa90c6 P 5cb9d430e4b94d578b436c1f9a9cafb8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 14:01:48.211091 11605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "cf637c0b31e34127b7b2599a8c2763a6"
I20240501 14:01:48.211158 11864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2420c2ca4eefa11a882651d7f9ac" candidate_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "662569f31808430ea81047cac005191e"
I20240501 14:01:48.211367 11864 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P 662569f31808430ea81047cac005191e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.211382 11605 raft_consensus.cc:3032] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:48.211531 11605 raft_consensus.cc:729] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf637c0b31e34127b7b2599a8c2763a6, State: Running, Role: LEADER
I20240501 14:01:48.210750 11472 leader_election.cc:288] T 5ffb2420c2ca4eefa11a882651d7f9ac P 5cb9d430e4b94d578b436c1f9a9cafb8 [CANDIDATE]: Term 5 election: Requested vote from peers 662569f31808430ea81047cac005191e (127.28.17.4:33167), cf637c0b31e34127b7b2599a8c2763a6 (127.28.17.2:42357)
I20240501 14:01:48.211864 11605 consensus_queue.cc:260] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "662569f31808430ea81047cac005191e" member_type: VOTER last_known_addr { host: "127.28.17.4" port: 33167 } } peers { permanent_uuid: "cf637c0b31e34127b7b2599a8c2763a6" member_type: VOTER last_known_addr { host: "127.28.17.2" port: 42357 } } peers { permanent_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8" member_type: VOTER last_known_addr { host: "127.28.17.1" port: 44355 } }
I20240501 14:01:48.212149 11605 raft_consensus.cc:3037] T 5ffb2420c2ca4eefa11a882651d7f9ac P cf637c0b31e34127b7b2599a8c2763a6 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:48.213083 11472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b316e036f734208a0cfab91895239b1"
dest_uuid: "5cb9d430e4b94d578b436c1f9a9cafb8"
from {username='slave'